Developing Quality Indicators for Family Support Services in Community Team-Based Mental Health Care

Abstract

Quality indicators for programs integrating parent-delivered family support services for children’s mental health have not been systematically developed. Increasing emphasis on accountability under the Affordable Care Act highlights the importance of quality-benchmarking efforts. Using a modified Delphi approach, quality indicators were developed for both program level and family support specialist level practices. These indicators were pilot tested with 21 community-based mental health programs. Psychometric properties of these indicators are reported; variations in program and family support specialist performance suggest the utility of these indicators as tools to guide policies and practices in organizations that integrate parent-delivered family support service components.

Introduction

Family to Family Support Services

Peer support involving lay community health workers is widely used across various health fields, such as asthma, diabetes, HIV and primary care, both within the United States as well as internationally (Chapman, Damio, Young, & Perez-Escamilla, 2004; Fisher, Earp, Maman, & Zolotor, 2009; Fisher et al., 2009; Lehmann & Sanders, 2007; Lewin et al., 2005; Pearson et al., 2007; Swider, 2002; Zuvekas, Nolan, Tumaylle, & Griffin, 1999). Within mental health, the movement to shift care towards a recovery orientation has spurred interest in hiring individuals with lived experience as peer support workers to assist consumers in this recovery process. The involvement of peer support workers in service delivery is often considered a sign of a system’s commitment to partnership and promotion of recovery (Richard, Jongbloed, & MacFarlane, 2009). In adult mental health, results from randomized trials have demonstrated client benefits associated with peer services, including increased contact with service providers, empowerment and recovery, and lower rates of re-hospitalization (Corrigan, 2006; Dixon et al., 2001; Sells, Davidson, Jewell, Falzer, & Rowe, 2006; Min, Whitecraft, Rothbard, & Salzer, 2007). The evidence base for peer-delivered (i.e., parent-delivered) services is more limited in children’s mental health (Hoagwood et al., 2010) but growing.

Within children’s mental health, family support services delivered by veteran parents have gained rapid momentum, with a quarter of states making such family support services a billable service through Medicaid or federal block grants (Center for Health Care Strategies Inc., 2012). These family support specialists (FSS; also known by a variety of other names including family partners, family support workers, parent partners, parent mentors, veteran parents,) are typically caregivers who have “lived experience” of parenting a child with mental health needs and are able to “give back” to other parents (Hoagwood, 2005; Koroloff, Elliott, Koren, & Friesen, 1996; Koroloff, Koren, Elliot, & Friesen, 1994; Osher, Penn, & Spencer, 2008). They provide a range of supportive services and their primary function is usually to model, coach, and empower parents in their journey to cope with, advocate and negotiate the fragmented children’s service systems. FSSs promote linkages to other families and to other children’s services and can decrease family isolation, increase a parent’s formal and informal support networks, and enable advocacy at the individual, systems, and policy levels. FSSs work in a variety of settings, sometimes as part of a provider team (i.e., embedded model), but historically, as community-based advocates working individually with a parent. When employed on teams, FSSs may serve as “translators” and facilitators of productive partnership between families and providers on the team.

The involvement of parents as family support providers represents a significant paradigm shift in children’s mental health services, where parents were traditionally viewed at best as passive recipients of services and at worst as the “cause” of their children’s problems. Since the mid-1980s when the federal government funded the Child and Adolescent Service System Program (CASSP), family participation has been viewed as an essential element in service quality (Stroul, 1996). A parent-delivered family support service represents a relatively new model of family support in children’s mental health. The use of veteran parents to assist other parents originated in the field of child health, with studies in asthma, diabetes, sickle cell anemia, and cystic fibrosis showing the benefits of such support in decreased parent strain (Ainbinder et al., 1998; Ireys, Chernoff, DeVet, & Kim, 2001; Ireys, Chernoff, Stein, DeVet, & Silver, 2001; Ireys, Sills, Kolodner, & Walsh, 1996; Sullivan-Bolyai et al., 2010; Sullivan-Bolyai, Knafl, Tamborlane, & Grey, 2004). Social support appears to directly affect parents’ own mental health and functioning and improves access to resources to ultimately influence child adjustment (Ireys et al., 2001a; Ireys et al., 2001b).

To date, the literature base on the efficacy of parent-delivered family support services remains sparse and direct evidence of rigorous studies on the effectiveness of parent-delivered family support services in mental health are slim (Blau et al., 2010; Hoagwood et al., 2010; Kutash, Duchnowski, Green, & Ferron, 2011). Further, there is no empirical support on how best to integrate such services within the array of services provided through a community-based organization. Limited data are available to guide services and policy implementation of this rapidly developing service model and workforce (e.g., Hoagwood et al., 2010; Robbins, et al., 2008).

To help promote standards and competency of the skills needed for this emerging profession, the National Federation of Families for Children’s Mental Health established a Certification Commission for Family Support (http://certification.ffcmh.org/). To this end, a list of content and competencies that can be considered for certification has been identified. While certification of FSSs identifies the minimum skills and experience required to function as a FSS, the policies and procedures needed by an agency or organization to support the function of FSSs do not yet exist. While there is broad consensus about tasks performed by FSSs (Obrochta, Anthony, Armstrong, Kalil, Hust, & Kernan, 2011; National Federation of Families for Children’s Mental Health, 2011), local variation in roles, responsibilities and practices associated with the FSS within an organization hinder an organization’s ability to adequately support and supervise this new workforce. In short, agencies or organizations adopting this service component have limited guidance on how best to assess and promote quality services to families in mental health care settings.

Healthcare Context: Why Develop Quality Measures?

The rapid expansion of Family Support Services nationally in the absence of clear consensus and evidence of effectiveness is cause for concern as the healthcare landscape is being shaped by the 2011 Affordable Care Act. That Act calls for the development of medical homes, accountable care organizations, and quality benchmarks to improve service quality while reducing health care expenditures. The viability of the FSS service model, like other behavioral services within the changing healthcare system, depends on the ability to identify components of quality care and link them to positive outcomes (Obrochta et al., 2011). The development of quality measures is a neglected and high priority for children’s mental health (IOM, 2000; IOM, 2006; Pincus, Spaeth-Rublee, & Watkins, 2011).

The rapid dissemination of evidence-based practices into community-based settings has focused attention on monitoring implementation quality (Fixsen, Naoom, Blasé, Friedman & Wallace, 2005). While quality-benching tools (including fidelity monitoring) are rare for community-based services in mental health, there are several strong examples including those by Bruns, Burchard, Suter, Leverentz-Brady, & Force (2004), Chorpita et al. (2010), Schoenwald, Ward, Henggeler, & Rowland (2000), and an even longer tradition for adults with serious mental illness (see Teague, Mueser & Rapp, 2012). As the research base on family support services grows, the availability of quality indicators to benchmark effective delivery of these services will be critical if this service model is to be sustained.

Study Context

Towards this end, this study examined the delivery of family support services by FSSs within an array of services provided through the Home and Community Based Services (HCBS) Waiver program. HCBS Waiver is an intensive, community-based program, serving children between five and 17 years of age (prior to their 18^th birthday), who have been classified as having serious emotional disturbances and who are at imminent risk for out of home placement. HCBS Waiver programs were selected as the focus of our study for several reasons. First, HCBS are federally approved programs authorized under Title XIX of the Social Security Act and these programs provide states flexibility to design and offer a mix of service options in order to meet the needs of specific groups of individuals who would otherwise require institutional care. Results from this study thus have some potential for generalization to other states and/or programs that offer similar services to similar populations. Second, these programs in many ways mirror a medical home model, whose key features include better access, care coordination, prevention, quality and safety (Sai, Tonniges, Osterhus, & Taba, 2004), and hence may be a model that could be supported within the Affordable Care Act in the healthcare reform era. Finally, family support services delivered by FSS is one of six Medicaid billable service options offered to families who are served, using a team-based approach for preventing out of home placements within New York State’s HCBS Waiver. Consequently the findings from this study may have direct applicability for policy and practice in New York and other states implementing similar services.

The goals of this study were two-fold: (1) Describe the development of a set of quality indicators (QIs) for HCBS Waiver programs at two levels: program level and individual FSS level. These QIs represent performance (rather than accountability) measures that can be used to determine the degree of adherence to a standard of practice standards. The QIs examined in this paper are consistent with recommendations for how mental health organizations and practitioners could participate in health care reform efforts to improve quality of care (Pincus et al., 2007). These indicators were developed using a well-established method in health care, the Delphi Method (Boulkedid, Abdoul, Loustau, Sibony, & Alberti, 2011); (2) Pilot test the QIs to assess their practical utility, using data from 21 HCBS programs in New York State. This paper provides a description of the variation in performance on these QI sets and explores the utility of these indicators for measuring quality of family support services within children’s mental health. A companion paper describes the relationship between these QIs and a measure of the organizational social context within these same programs (see Olin et al., this issue).

Methodology

This study involves a two-part approach: development of QIs and assessing the utility of these QIs using data previously collected from 21 HCBS programs. QI development occurred independently of the data collected to assess the content and process of family support services, which represented a separate aspect of this study (and are discussed in Wisdom et al., 2013, this issue).

Modified Delphi Method for Identifying Quality Indicators

The Delphi technique is a highly structured process that employs a series of “rounds” to gather and compile information from expert stakeholders in order to gain consensus regarding a complex phenomenon (Boulkedid et al., 2011; Linstone & Turoff, 1975). In this study, a modified Delphi technique was used to establish indicators for both programs (i.e., policies and procedures needed to implement a high quality family support service component) and FSSs (i.e., activities that reflect high quality FSS practices). This approach was used to support the validity of the content of indicators being established.

Development of Initial Quality Indicators

Because no empirical literature exists on the critical standards needed to deliver high quality FSS services within children’s mental health services, our research team relied on a review of available documents that described competencies and guidelines for FSS activities. These included: National Federation of Families for Children’s Mental Health’s Guidelines and Certification Domains (http://certification.ffcmh.org/), the New York State Office of Mental Health’s draft guidelines for family-to-family support activities, New York State HCBS Waiver family support services worker competencies guidance documents (http://www.omh.ny.gov/omhweb/guidance/hcbs/competencies/family_support_services.html), the Parent Empowerment Program Training Manual that serves as New York State Office of Mental Health’s state early-entry training standard for FSSs working in OMH licensed organizations (Jensen & Hoagwood, 2008; PEP team, 2012).

Based on a thorough review of these documents, an initial list of indicators were developed and selected using the following criteria: indicators should be (1) malleable through intervention; (2) understandable to a broad audience (i.e., high face validity), (3) relevant to key stakeholders, and (4) observable or reportable. An initial list of indicators was developed to delineate program policies and procedures necessary for a program to support the delivery of a high quality service (Program QIs) and an initial list of FSS activities that would be reflective of high quality family support services (FSS QIs).

Expert Panel

Twenty-four individuals were selected by the research team for their familiarity and specific experiences with family support services; these individuals represented three different stakeholder groups: (1) researchers with specific expertise in family support services, (2) clinicians, providers or policy makers involved in child mental health programs that provide family support services, and (3) FSSs with national or state advocacy experience. Eighteen individuals were able to participate in the first round of the consensus building process and 11 were able to participate in the second round. The 18 participants were evenly distributed among the three stakeholder groups with 33% representing researchers, 39% service providers or policy makers, and 28% FSSs. The experts were predominantly female (82%) and 50% identified themselves as having had experience as caregivers of a child with special needs. The panel was very experienced; researchers reported an average of 23 years (SD = 7.54) of experience in the area, service providers and policy makers reported an average of 12 years of experience (SD = 4.60), and FSSs reported an average of 16 years (SD = 5.29). Individual rater response was kept anonymous. The results from both rounds were aggregated and/or de-identified before being presented to members of the Study Advisory Board in a face-to-face meeting to make final determinations on any indicators that lacked clear consensus. Eleven members of the Study Advisory Board participated as one of the 18 individuals involved in the Delphi process.

Procedure for Consensus Development

The initial list of indicators (17 Program QIs and 28 FSS QIs) was distributed to the expert panel through email. Each expert panel member was asked to rate each item as to its importance or appropriateness as an indicator reflecting high quality FSS services (that is, reflecting best practices). The rating could range from 1 to 9, with a rating of 1 reflecting a best practice or extremely appropriate quality indicator, a rating of 4 reflecting an equivocal rating and a rating of 9 reflecting an extremely inappropriate practice. Raters were asked to provide their ratings based on ideal circumstances and not on the feasibility of implementation or on their own community experiences. Raters were also asked to provide a rationale for their ratings, suggest indicators not included on the list and/or suggest improved wording in order to assist in the consensus building process. The nine ratings were collapsed into 3 categories to aid in determining consensus among experts. Ratings between 1 and 3, 4 and 6, and 7 and 9 were collapsed and labeled Appropriate, Equivocal, and Inappropriate, respectively. An indicator was considered to have achieved consensus when at least 80% of the raters rated the indicator in the same category.

After the first round of ratings from the expert panel, those indicators that did not achieve 80% consensus were eliminated or revised to either increase clarity or to reduce content and concept overlap based on comments from raters. Revised indicators were distributed to the expert panel to be rated. After the second round of ratings, indicators that still did not reach consensus were discussed by members of the Study Advisory Board until consensus was reached. Only two items among both the Program and FSS indicators were reviewed by members of the Study Advisory Board after Round Two. These items are discussed below in the Results section. This process resulted in 14 Program QIs and 27 FSS QIs (21 representing appropriate FSS practices and 6 representing inappropriate FSS practices).

Pilot test of Quality Indicators with 21 HCBS Waiver Programs

The utility of these QIs were assessed by applying them to data previously collected from 21 HCBS Programs as follows:

Programs in pilot testing

All 33 OMH-funded HCBS Waiver Programs in New York State that serve children with serious emotional disturbances were invited to participate in this study. Three programs were not eligible to take part in the study because they either did not have at least a half-time FSS or an FSS on staff for at least 6 months. Twenty-one (64%) programs participated in this study. These 21 programs did not differ from the rest of the non-participating HCBS programs across the state in terms of program capacity (number of families that can be served ranged from 12–144; mean = 48.9, SD = 37.5) and region of the state, with at least half the programs in each of the five regions participating in the study; participation rates were highest in the Western (and most rural) regions of the state. The majority of the FSSs who took part in the study also did not differ from the FSSs employed in this type of program across the state. They had at least a high school degree (71.5%), 19.1% had a bachelor’s degree or some graduate education; one had a Master’s degree in social work. Approximately three-quarters were credentialed through either the Parent Empowerment Program (47.6%) or had Family Development Credentials (23.8%). The majority (71.4%) had been with the program between 1–5 years.

Information obtained for Coding Quality Indicators

Two sources of data were used to score each program on the Program QIs and FSS QIs. All data were collected between August and October 2011. For Program QIs, transcripts of audiotapted interviews (conducted either by phone or face-to-face) with the Program Director, one Individualized Care Coordinator and one FSS at each program were obtained. The Program Director completed a modified MacArthur’s Survey (Schoenwald et al., 2008) including questions focused on fiscal issues, the case referral process, staff supervision and training opportunities, and staff roles and interactions. Each of the three respondents was asked questions about what services FSSs provide, how the FSSs work with families, and how they work with both internal Waiver and external staff.

To obtain information about the FSS practices needed to determine performance on the FSS QIs, transcripts of audiotapes using a Standardized Walkthrough procedure of the first two sessions with a FSS from each program was used. This Standardized Walkthrough procedure is an in-vivo method of data collection used widely in medical training and business and management circles (Carney, Dietrich, Eliassen, Owen, & Badger, 1999; Ford et al., 2007; Gustafson, 2004; Horsky, Kaufman, Oppenheim, & Patel, 2003). The procedure for this study was developed and described in an earlier study (Wisdom, Olin, Shorter, Burton, & Hoagwood, 2011). A member of the research team who is a FSS, assumed the role of the parent, “Katherine,” in two face-to-face meetings with FSSs. No deception was used; FSSs were aware that the “parent” was a member of the research team. The parent presented a standardized scenario in which she was a single mother, with three children, one of whom was enrolled in the Waiver program. The standardized scenario included several pressing issues that were reported to the FSS during the two meetings including the child’s problems at school, an upcoming court date, safety issues, concern for the parent’s other children (two younger daughters), and for the parent’s job.

Scoring the Quality Indicators

Two members of the research team were trained to review and rate transcripts of audiotapes according to assigned benchmarks of performance: programs and FSSs were rated as either as performing below (scored as 1), meeting (scored as 2) or exceeding expectations (scored as 3), with specific criteria developed to guide the rating of each indicator. For Program QIs, the final rating for each indicator was the lowest score across the three respondents. For FSS QIs, each walkthrough transcript was reviewed to rate FSS performance; QIs representing inappropriate practices were reverse scored (as −1). Each coder independently rated the staff interviews and FSS walkthrough interviews for each program. To facilitate inter-rater reliability, coders met at the end of rating each program (for both Program and FSS QIs) to check for reliability and discuss discrepancies; discrepancies in ratings were resolved through consultation with the first author to arrive at a consensus. There was very good reliability across the two trained raters, averaging 98% with a range of 88%–100% (kappa=.96) for Program QIs, and 96% with a range of 90–100% (kappa=.90) for FSS QIs.

Results

Establishing Consensus on Quality Indicators

Consensus was achieved for each indicator when 80% or more of the expert raters placed the indicator in the same category. For those indicators that reached consensus during the first and second round of ratings, they reached consensus with ratings in either the best practices or extremely appropriate practice category (i.e., scored between 1 and 3) or the extremely inappropriate practice category (i.e., scored between 7 and 9) with no indicator placed in the equivocal category. The six FSS QIs that were rated in the inappropriate practice range were separated from the other indicators and labeled Inappropriate Practices in the final indicator set.

Program QIs

Of the 17 initial Program QIs distributed to the expert panel in Round 1, 12 (71%) of the items met the 80% consensus criteria for inclusion. Five QIs with divergent ratings were revised, combined or deleted to improve clarity or reduce overlap and one new indicator was added based on expert rater suggestion. In Round 2, three indicators were presented to raters. One indicator met consensus criteria and the two divergent indicators were deferred for discussion in a face-to-face meeting by members of the Study Advisory Board. Members of the Study Advisory Board suggested the addition of one new indicator and integration of the two divergent indicators with an existing indicator due to content overlap. See Table 1 for a list of the final list of 14 Program QIs with indicators categorized according to the following domains: Role, Fiscal, Standards to Guide Practice, and Structural.

Table 1.

List of Program Quality Indicators

A program that provides high quality family support services includes…

ROLE

1.Clear roles and responsibilities for FSSs (e.g., detailed job descriptions available) 2.Staff that clearly understand the role of the FSS within the goal of the program

FISCAL

3.A specific budget allocated for family support services 4.Program flexibility in use of family support services even when such services are not billable (e.g., pre-enrollment into program and post discharge) 5.Employment benefits provided (e.g., health, vacation) to FSSs

STANDARDS TO GUIDE PRACTICE

*6.A standardized protocol or framework used by program staff with all families to guide intensity, type and progress of family support services relative to family needs, goals and strengths 7.FSSs that have received training or certification in core competencies 8.FSSs employed with experience of parenting a child with emotional, developmental, behavioral, substance use or mental health concerns

STRUCTURAL

9.Specialized supervision/consultation structures in place to help integrate FSS role on the team and with families 10.Structures in place to facilitate teamwork among FSSs and other staff on team (e.g., team meetings) 11.Informal communications among team members outside of planned or structured meetings (e.g., phone calls, discussions in passing) 12.Experienced family members (non-employees) as part of a board within agency 13.Program flexibility to allow integration of new/innovative ideas from staff 14.Program integration of the FSSs as equal members and/or active members of the team in working with a family

^*Item that required additional work to reach consensus

The Program QI that elicited the most divergent rating and generated the most debate in the face-to-face meeting of Study Advisory Board members involved the use of assessment tools or standard protocols to determine the need for family support services (61% of the expert panel members agreed in Round 1 and 72% of the expert panel members agreed in Round 2.). The lack of 80% consensus in both rounds was driven primarily by ratings in the equivocal range, indicating the uncertainty of raters about including the use of assessment tools or standardized protocols as program quality indicators. Experts expressed concerns about use of assessment tools or protocols to determine need or level of care as this is not usual in family support services and not in line with the flexibility associated with family-centered care, which emphasizes family determination of need rather than program or other determination. Concerns were expressed about the use of an assessment tool as a gatekeeper or a screen for service eligibility for family support services and the importance of basing determination on family functioning and not on child functioning. In contrast, some experts considered the use of assessment tools or protocols to guide services as empowering and advancing better practices in working with caregivers. This indicator achieved consensus after much discussion, and was revised and combined with another indicator, to yield the following: A standardized protocol or framework is used by program staff to guide determination of the level of family support services throughout service use (Program QI #6).

FSS QIs

Of the initial 28 FSS QIs sent to Expert Panel members in Round 1, 22 (79%) met the 80% consensus criteria; of these, four were combined into 2 other QIs to improve clarity and reduce overlap in content, yielding a final number of 20 QIs at the end of Round 1. Six QIs with divergent ratings were revised for clarity and one new QI was added based on feedback from members of the expert panel. Seven indicators were presented to the expert panel members in Round 2. Of these seven indicators, five met criteria for inclusion, while two indicators continued to show divergent ratings and were deferred for discussion by members of the Study Advisory Board.

Similar to the Program QIs, the two FSS QIs that elicited the most divergent ratings and generated the most debate among the Study Advisory Board members involved the use of standardized assessment methods to identify family problems or track family’s progress towards goals. Expert raters expressed concerns due to the current limitation of standardized assessments such as the validity, applicability and even existence of appropriate assessments for use with family support services. Concerns were expressed about standard assessments being “too clinical” and contrary to a family-centered approach and may turn off families and advocates. Consensus was reached about working with families systematically and collaboratively to assess or evaluate progress. Both indicators achieved consensus after discussion by the members of the Study Advisory Board; and the indicators were revised as: Works with the family to systematically identify family goals or needs (FSS QI #10); and systematically monitors family's progress towards goals (FSS QI #11). See Table 2 for a list of the final 27 FSS Level QIs categorized in six domains representing Role, Relationship with Team, Ethics, Knowledge, Skill, and Family Engagement, and one domain representing inappropriate FSS level practices.

Table 2.

List of FSS Quality Indicators by Domain

A skilled FSS…..

ROLE

1.Describes FSS role and what they do in relationship to other team members. 2.Describes or demonstrates boundaries of own roles and responsibilities toward families

RELATIONSHIP WITH TEAM

3.Demonstrates positive regard for the role of other staff on team 4.Supports the development of more effective partnerships between family and other members of the team.

ETHICS

5.Discusses how information they learn is shared among staff and outside family. 6.Establishes a quiet, confidential location to talk (whenever possible at location of caregiver’s choice, e.g., home, school, diner, etc.).

KNOWLEDGE

7.Demonstrates knowledge of community supports and resources.

SKILLS IN PRIORITY SETTING

8.Knows how and when to facilitate goal oriented and solution focused problem solving to help the family accomplish their goals. 9.Identifies safety concerns and existence of safety plan; works with family and team to initiate a plan as appropriate and/or address the adequacy of existing plan. *10.Uses systematic and standardized methods to work collaboratively with the family to identify goals of family support services based on family strengths and needs *11.Uses systematic and standardized methods to work collaboratively with the family to monitor the progress of goals of family support services. 12.Facilitates a family's identification of priorities and concerns.

SKILLS IN PROVIDING LINKAGES

13.Works with family to identify, reconnect and or build their formal and informal support system.

SKILLS IN PROVIDING EMOTIONAL SUPPORT

14.Facilitates caregiver identification of ways to promote self-care. 15.Uses his or her own experiences to support and or normalize a caregiver’s experience and promote hope

SKILLS IN EDUCATING

16.Promotes empowerment by targeting assistance to caregiver need. 17.Exercises sound judgment when providing advice or recommendations. 18.Models or coaches caregiver use of new skills (e.g. through role plays).

FAMILY ENGAGEMENT

19.Promotes family voice and choice through shared decision making (e.g., helps family articulate cultural, spiritual and/or religious values and preferences). 20.Reframes or clarifies a caregiver's perspective or position in a way that avoids criticism or judgment of caregiver. 21.Uses strength-based language.

An inappropriate action of a family peer advocate is one that….

FAMILY ENGAGEMENT

22.Uses communication that indicates blame or criticism of caregiver. 23.Provides advice or service that is beyond the scope or role of FSS (e.g., legal or medical advice, transportation, providing child care). 24.Uses deficit-based language. 25.Uses medical jargon inappropriately (e.g., in a patronizing way). 26.Is directive and makes decisions independent of the caregiver about what is good for the family.

RELATIONSHIP WITH TEAM

27.Works with a caregiver to go around decisions of the rest of the team.

^*Items that required additional work to reach consensus

Performance of the Quality Indicators

In the following section, we describe how both sets of QIs performed at the program level, as well as how each individual indicator performed across programs. These analyses were conducted to assess the utility of these indicators in detecting variation in program performance and their potential for use as benchmarking tools.

Analysis of QIs at the Program Level

Program QIs

Among the 14 Program QIs, two were eliminated from the analyses because of excessive “Don’t Knows” (i.e., insufficient information was provided for the coders to use in rating the indicator). Eight programs (38%) were rated as having met expectations on half or more of the 12 program indicators, an additional 8 programs (38%) were rated as not meeting expectations on at least half of the program indicators and no programs exceeded expectations on more than half of the indicators. For the remaining 5 programs, 4 were rated as having a similar number of not met, met and exceeding expectations; 1 program was rated as having met expectations on half of the indicators and below expectations on the other half of the indicators. Generally, these 21 programs appear to be divided between meeting and not meeting the majority of program quality indicators, see Table 3.

Table 3.

HCBS Waiver Program Performance on Program and FSS Quality Indicators

Program QIs					FSS QIs
# of Items rated Below	# of items rated Meets	# of items rated Exceeds	# of items rated Don’t Know	Sum Program Level QI (n=12)*	# of items rated Below	# of items rated Meets	# of items rated Exceeds	Total FSS QI (n = 27)		Total Program & FSS QI
								Approp. (n=21)	Inapprop.. (n=6)
5 (42%)	4 (33%)	2 (17%)	1 (8%)	19	1 (5%)	7 (33%)	13 (69%)	54	0	73
1 (8%)	6 (50%)	5 (42%)	0 (0%)	28	5 (24%)	13 (69%)	3 (14%)	40	0	68
3 (25%)	4 (33%)	5 (42%)	0 (0%)	26	3 (14%)	6 (29%)	12 (57%)	51	0	77
3 (25%)	4 (33%)	5 (42%)	0 (0%)	26	8 (38%)	4 (19%)	9 (43%)	43	−1	68
4 (33%)	7 (58%)	0 (0%)	1 (8%)	18	6 (29%)	14 (67%)	1 (5%)	37	−2	53
3 (25%)	7 (58%)	2 (17%)	0 (0%)	23	6 (29%)	10 (48%)	5 (24%)	41	0	64
4 (33%)	7 (58%)	0 (0%)	1 (8%)	18	4 (19%)	10 (48%)	7 (33%)	45	0	63
3 (25%)	5 (42%)	4 (33%)	0 (0%)	25	6 (29%)	6 (29%)	9 (43%)	45	0	70
3 (25%)	6 (50%)	3 (25%)	0 (0%)	24	12 (57%)	8 (38%)	1 (5%)	31	0	55
3 (25%)	6 (50%)	3 (25%)	0 (0%)	24	4 (19%)	4 (19%)	13 (69%)	51	0	75
6 (50%)	5 (42%)	0 (0%)	1 (8%)	16	7 (33%)	11 (52%)	3 (14%)	38	−2	52
8 (67%)	3 (25%)	0 (0%)	1 (8%)	14	18 (86%)	3 (14%)	0 (0%)	24	−3	35
4 (33%)	6 (50%)	2 (17%)	0 (0%)	22	4 (19%)	6 (29%)	11 (52%)	49	0	71
9 (75%)	3 (25%)	0 (0%)	0 (0%)	15	6 (29%)	12 (57%)	3 (14%)	39	0	54
3 (25%)	5 (42%)	4 (33%)	0 (0%)	25	9 (43%)	9 (43%)	3 (14%)	36	−3	58
9 (75%)	2 (17%)	0 (0%)	1 (8%)	13	17 (81%)	3 (14%)	1 (5%)	26	−4	35
9 (75%)	3 (25%)	0 (0%)	0 (0%)	15	14 (67%)	7 (33%)	0 (0%)	28	−2	41
8 (67%)	4 (33%)	0 (0%)	0 (0%)	16	8 (38%)	5 (24%)	8 (38%)	42	0	58
6 (50%)	6 (50%)	0 (0%)	0 (0%)	18	5 (24%)	5 (24%)	11 (52%)	48	0	66
9 (75%)	2 (17%)	0 (0%)	1 (8%)	13	9 (43%)	11 (52%)	1 (5%)	34	0	47
6 (50%)	5 (42%)	0 (0%)	1 (8%)	16	6(29%)	13 (69%)	2 (10%)	38	0	54

^*Excluded 2 QIs (items 1 and 13) from the program level analysis because of excessive DKs (unable to code these on 75% of programs), therefore means are based on 12 indicators. Ratings of below scored as 1, meets scored as 2 and exceed scored as 3.

FSS QIs

For the FSS QIs, 5 programs (24%) were rated as exceeding expectations on half or more of the indicators, 6 (28.5%) of the programs were rated as meeting expectations on half or more of the indicators while 4 (19%) were rated as not meeting expectations on at least half the FSS QIs. The remaining 6 programs (28.5%) had similar numbers of indicators rated as below, at and above expectations on FSS QIs. These results suggest considerable variation in quality of FSS practices in these 21 programs.

The mean program performance score on the 12 Program QIs was 19.71 (SD=4.87) out of a total score of 36 (range 13 to 28). The mean performance score on the 27 FSS QIs (n= 21, plus 6 inappropriate indicators that were reverse scored) was 40.00 (SD=8.35) out of a total score of 63 (range 21 to 54). Across the 21 HCBS Waiver Programs, the total Program and FSS QI score averaged 58.90 (SD=12.32) out of a total score of 99 (range 35–77). These results suggest that the 21 HCBS Waiver Programs show variation in Program and FSS QI ratings, with no ceiling effects uncovered; suggesting there is room for growth and positive change in each set of indicators. Notably, HCBS Waiver Program performance on Program QIs and FSS QIs were highly correlated among these 21 programs (Spearman’s rho = 0.53, p = 0.013).

Analysis of Individual Indicators

Program QIs

Three Program QIs were rated as below expectations for more than 75% of the programs. Ninety-five percent of the programs were below expectations in using a standardized protocol to guide intensity, type and progress of family support services (Program QI # 6); 86% either did not have an advisory board or a family member as part of the board (Program QI #12); and 76% of the programs were below expectations for having specialized supervision/consultation structures in place for FSSs to help maintain their role on the team and with families (Program QI #9).

Four Program QIs were rated as meeting expectations for the majority of the programs. These include employing a FSS with a special needs child (Program QI #8, 86%), employment benefits for FSS (Program QI #5, 67%), having FSSs trained in core competencies (Program QI #7, 62%) and having a specific budget allocated for FSS services (Program QI # 3, 57%).

The most common indicator where Waiver programs exceeded expectations was the indicator that rated staff as clearly understanding the role of the FSS within the goal of the program (Program QI #2, 38%), see Table 4.

Table 4.

HCBS Waiver Program Performance on Program QIs: Number and % Below, Meeting, or Exceeding Expectations

	% Below Expectations (Rating of 1)	% Meets Expectations (Rating of 2)	% Exceeds Expectations (Rating of 3)	% programs rating is missing (“DK”) raters unable to determine	Mean (SD) rating across all Programs (Mean calculated with DKs removed) Range 1–3
ROLE
Indicator #1 Clear role for FSS	Unable to Determine	Unable to Determine	0%	76% (16/21 )	1.4 (0.55)
Indicator #2 Staff understand role of FSSs	9 (43%)	4 (19%)	8(38%)	0%	2.0 (.92)
FISCAL
Indicator #3 Budget for FSSs	4 (19%)	12 (57%)	0(0%)	24% (5/21)	1.8 (.45)
Indicator #4 Flexible even if not billable	8 (38%)	8 (38%)	5 (24%)	0%	1.9 (.79)
Indicator #5 Employment benefits	4 (19%)	14 (67%)	2 (10%)	1 (5%)	1.9 (.55)
STANDARDS TO GUIDE PRACTICE
Indicator #6 Standardized protocol/framework	20 (95%)	0 (0%)	1 (5%)	0%	1.1 (.44)
Indicator #7 Training in core competencies	8 (38%)	13 (62%)	0 (0%)	0%	1.6 (.50)
Indicator #8 FSS parent of child with ED etc.	1 (5%)	18 (86%)	0 (0%)	2 (10%)	2.0 (.23)
STRUCTURAL
Indicator #9 FSS supervision structure	16 (76%)	3 (14%)	2 (10%)	0 (0%)	1.3 (.66)
Indicator #10 Structures to facilitate teamwork	7 (33%)	11 (52%)	3 (14%)	0 (0%)	1.8 (.68)
Indicator #11 Informal communications with team occur including FSSs	6 (29%)	8 (38%)	7 (33%)	0 (0%)	2.0 (.80)
Indicator #12 Family part of advisory board	18 (86%)	1 (5%)	2 (10%)	0 (0%)	1.2 (.62)
Indicator #13 Program open to new ideas from staff	Unable to Determine	Unable to Determine	0%	76% (16/21)	1.4 (0.55)
Indicator #14 FSS as equal team members	8 (38%)	8 (38%)	5 (24%)	0 (0%)	1.9 (.79)

FSS QIs

Four FSS QIs were rated as below expectations for the majority of FSSs: systematically monitoring progress towards goals (FSS QI #11, 95%), using systematic and standardized methods to work collaboratively with the family to identify goals of family support services (FSS QI #10, 86%), how information they learn is shared (FSS QI #5, 67%), and facilitating a family's identification of priorities and concerns (FSS QI #12, 52%). Six FSS QIs were rated as meeting expectations for the majority of FSSs: demonstrating boundaries with families (FSS QI #2, 67%), establishing a confidential location (FSS QI #6, 62%), demonstrating positive regard for the role of other staff on team (FSS QI #3, 52%), building support systems (FSS QI #13, 52%), promoting caregiver self-care (FSS QI #14, 52%), and using strength-based language (FSS QI #21, 52%). The most common indicators that were rated as exceeding expectations by FSSs were: ability to reframe a caregiver's perspective to avoid criticism or judgment of caregiver (FSS QI #29, 48%), exercise sound judgment (FSS QI 16, 48%) and promote empowerment (FSS QI #15, 43%).

Only seven (33%) of the 21 FSSs rated scored as exhibiting any of the six inappropriate practices. Two indicators were rated as not occurring with any of the FSSs rated (FSS QI #25: using medical jargon and FSS QI #27: works to go around the decisions of the rest of the team). The most commonly rated inappropriate indicator was the use of communication that indicates caregiver blame (FSS QI #22, 29%) (See Table 5).

Table 5.

HCBS Waiver Program Performance on FSS QIs: Number and % Below, Meeting, or Exceeding Expectations

	% Below Expectations (Rating of 1)	% Meets Expectations (Rating of 2)	% Exceeds Expectations (Rating of 3)	Mean (SD) across all Programs
ROLE
Indicator #1 Described FSS role and what they do in relationship to other team members	10 (48%)	8 (38%)	3 (14%)	1.7 (.73)
Indicator #2 Describes or demonstrates boundaries of own roles and responsibilities towards families	6 (29%)	14 (67%)	1 (5%)	1.8 (.54)
RELATIONSHIP WITH TEAM
Indicator #3 Demonstrates positive regard for the role of other staff on team	4 (19%)	11 (52%)	6 (29%)	2.1 (.70)
Indicator #4 Supports the development of more effective partnerships between family and other members of team	9 (43%)	7 (33%)	5 (24%)	1.8 (.81)
ETHICS
Indicator #5 Discusses how information they learn is shared among staff and outside family	14 (67%)	3 (14%)	4 (19%)	1.5 (.81)
Indicator #6 Establishes a quiet, confidential location to talk	1 (5%)	13 (62%)	7 (33%)	2.3 (.56)
KNOWLEDGE
Indicator #7 Demonstrates knowledge of community supports and resources	4 (19%)	9 (43%)	8 (38%)	2.2 (.75)
SKILL PRIORITY SETTING
Indicator #8 Knows how and when to facilitate goal oriented and solution focused problem solving to help the family accomplish their goals	8 (38%)	9 (43%)	4 (19%)	1.8 (.75)
Indicator #9 Identifies safety concerns and existence of safety plan; works with family and team to initiate one as appropriate and/or address the adequacy of existing plans	9 (43%)	7 (33%)	5 (24%)	1.8 (.81)
Indicator #10 Uses systematic and standardized methods to work collaboratively with the family to identify goals of family support services based on family strengths and needs	18 (86%)	2 (10%)	1 (5%)	1.2 (.51)
Indicator #11 Uses systematic and standardized methods to work collaboratively with the family to monitor the progress of goals of family support services	20 (95%)	0 (0%)	1 (5%)	1.1 (.44)
Indicator #12 Facilitates a family's identification of priorities and concerns	11 (52%)	6 (29%)	4 (19%)	1.7 (.80)
SKILL-LINKAGE
Indicator #13 Works with family to identify, reconnect and/or build their formal and informal support system	6 (29%)	11 (52%)	4 (19%)	1.9 (.70)
SKILL-EMOTIONAL SUPPORT
Indicator #14 Facilitates caregiver identification of ways to promote self-care	4 (19%)	11 (52%)	6 (29%)	2.1 (.70)
Indicator #15 Uses his/her own experience to support/normalize a caregiver's experiences and promote hope	4 (19%)	9 (43%)	8 (38%)	2.2 (.75)
SKILL-EDUCATION/SKILL
Indicator #16 Promotes empowerment by targeting assistance to caregiver need	4 (19%)	8 (38%)	9 (43%)	2.2 (.77)
Indicator #17 Exercises sound judgment when providing advice or recommendations	4 (19%)	7 (33%)	10 (48%)	2.3 (.78)
Indicator #18 Models or coaches caregiver use of skill (e.g., through role plays)	7 (33%)	8 (38%)	6 (29%)	2.0 (.80)
FAMILY ENGAGEMENT
Indicator #19 Promotes family voice and choice through shared decision making	9 (43%)	6 (29%)	6 (29%)	1.9 (.85)
Indicator #20 Reframes or clarifies a caregiver's perspective or position in a way that avoids criticism or judgment of caregiver	4 (19%)	7 (33%)	10 (48%)	2.3 (.78)
Indicator #21 Uses strength-based language	2 (10%)	11 (52%)	8 (38%)	2.3 (.64)
INAPPROPRIATE FSS Indicators
	Number and % of FSSs that did not exhibit this practice (rating of 0)	Number and % of FSSs that exhibited inappropriate practice (Rating of −1)	Mean (SD) rating across all programs Range −1−0
FAMILY ENGAGEMENT
Indicator #22 Uses communication that indicates blame or criticism of caregiver	15 FSSs (71%)	6 (29%)	−0.29(.46)
Indicator #23 Provides advice or service that is beyond the scope or role of FSS	17 FSSs (81%)	4 (19%)	−0.19 (.40)
Indicator #24 Uses deficit-based language	17 FSSs (81%)	4 (19%)	−0.19 (.40)
Indicator #25 Uses medical jargon inappropriately	21 FSSs (100%)	0	0 (0)
Indicator # 26 Is directive and makes decisions independent of the caregiver about what is good for the family	18 FSSs (86%)	3 (14%)	−.14 (.36)
RELATIONSHIP WITH TEAM
Indicator # 27 Works with a caregiver to go around decisions of the rest of the team (e.g., splitting)	21 FSSs (100%)	0	0 (0)

Overall, the variation of HCBS Waiver program and FSS performance on each indicator suggests good sensitivity of each of these indicators in detecting variation in performance. These QI indicator sets have potential to be specific and sensitive measures for benchmarking Program Level and FSS Level service quality in these Waiver Programs.

Discussion

This is the first attempt to generate consensus from a group of child and family mental health experts for the purpose of developing quality indicators for family support services. Quality metrics are gaining ascendancy within health care as a means of promoting safe, effective, timely, efficient, equitable and patient-centered health care (IOM, 2001). QIs are an essential element of system accountability.

Both Program and FSS QIs were developed through a modified Delphi approach, suggesting that experts representing different perspectives (researchers, providers/policy makers, and FSSs) can arrive at an overall agreement on a set of indicators to guide standards of practice. It is worth highlighting that difficulties in achieving consensus on several key indicators revolved exclusively around the use of standardized assessments or protocols to guide both program policies and FSS practices. Such difficulties likely reflect gaps in knowledge as well as philosophical tensions in the field that exist around certain standards of practice, particularly the potential for misuse of standardized assessments and protocols. Concerns were expressed about the potential use of these kinds of data to curtail services or limit access. However, to date, there was also recognition that the lack of standardized or well-validated measures to assess family support practices beyond service satisfaction is constraining growth of this service model (see Hoagwood et al., 2010). Experts’ uncertainty about these quality indicators also mirrors tensions in the field around whether such services, targeted at caregivers (rather than at the child specifically) are needed and should be paid for, when evidence of its impact on child outcomes is still limited. Some of these tensions may be resolved as more fully integrated models for children, adolescents, families and adults are provided as part of a continuum of services. While the use of standardized measures or systematic methods to guide practice may be philosophically at odds with a family-centered approach that values individualization and choice of services, in the end, there was agreement that standard methods for assessing and monitoring the impact of family support practices that are based on collaboration between the FSS and family are an essential element. Given the nature of family support services, this type of collaborative assessment system may need to be a key aspect of any assessment tool used to systematically monitor family progress; the creation of such an assessment protocol is critical to adequately capture the unique aspects of family support services.

Pilot testing of these quality indicators with 21 HCBS Waiver Programs in New York State supports the potential utility of these program level and individual FSS level QIs for assessing standards of practice. These indicators demonstrated adequate psychometric properties, with good face validity, high inter-rater reliability across trained coders. There was substantial variation in how programs performed as well as individual indicator performance across programs. Such variation suggests that differentiation of programs is possible and points to the potential utility of these metrics for improving quality.

With respect to Program QIs, the indicator related to the use of standardized protocols to guide family support services was the most problematic for programs. All programs (with one exception) failed to meet expectations on this indicator. While admission to HCBS Waiver programs are guided by specific eligibility criteria, access to individual service components (including family support services) once admitted, are not guided by specific criteria. The lack of standardized methods to guide ongoing service delivery is not uncommon in mental health care. Another QI that was problematic for the HCBS programs in this study was the lack of specialized supervision/consultation structures in place for FSSs. FSSs are typically supervised by the Program Director or the Individual Care Coordinator, and such supervision typically revolves around administrative and clinical issues, rather than FSS role-specific responsibilities. FSSs have cited such challenges in role differentiation and value of role by their programs as an impediment to their effectiveness, both within their provider teams as well as with families they serve (see Kutash, Acri, Pollock, Armusewicz, Olin, & Hoagwood, this issue).

With respect to individual FSS QIs, the performance of the majority of the FSSs was below expectations on four of the 27 indicators. In particular, the QIs related to monitoring progress towards goals and using systematic and standardized methods to identify goals were below expectations. These challenges are quite consistent with program level challenges in having standardized protocols for monitoring service delivery to clients. The majority of FSSs were also below expectations in discussing how they share information on what they learn with other staff and in facilitating a family's identification of priorities and concerns. Given the unique positions of FSSs and their role as cultural brokers for both the provider teams and the family, how family priorities, needs and information are shared with all team members, including the family, are significant issues.

Notably, performance on Program QIs and FSS QIs were highly correlated among these 21 programs, suggesting that changes in program policies and practices may have the potential to also influence individual FSS practices. More research with the use of these indicators is needed to establish their utility and generalizability across different types of service settings.

Limitations

First, although many indicators achieved high levels of consensus among experts, full consensus around several program and FSS level indicators were achieved only after two rounds and much debate, indicating differences and uncertainties among experts and suggesting that these indicators be used as guidelines rather than as a set of rigidly applied benchmarks. Second, these indicators were developed in the absence of clear evidence about the effectiveness of parent-delivered family support services. This field is still being developed and the research base on the effectiveness of FSS remains to be established. Thus, unlike other areas of health where clear models for effective interventions exist, for parent-delivered family support, the data are suggestive but not definitive. Nevertheless, we believe development of quality indicators is an important step to align this service model with the new healthcare emphasis on accountability, competencies and performance standards.

Third, these indicators were tested within a service model where FSSs are embedded as part of a larger service program. Despite this, we suspect that many of the FSS level indicators may be generalizable because they pertain to specific skills and competencies of FSSs that are applicable across service models. Fourth, these Program Level and Individual FSS level indicators represent the current views of expert stakeholders and we can expect that these would evolve as more research and knowledge develops with this expanding service model within children’s mental health.

Fifth, performance on these QIs may be influenced by the procedures used in this study. Specifically, data used to assess the utility of these QIs were independently collected without reference to these QIs; further, the FSS QIs were rated based only on the first two sessions of a Standardized Walkthrough with the FSSs. It is thus possible that ratings may vary as much from a lack of coverage of relevant areas in the interviews or the walkthrough procedure as actual differences in performance areas. Further, because no deception was used as part of the Standardized Walkthrough, FSSs may be conscious of being evaluated, such that these scores may reflect higher performance than might occur in authentic interactions. Despite this limitation, performance on these QIs showed significant diversity, suggesting the sensitivity of this procedure for capturing a range of practices exhibited by FSSs in this study.

Finally, ways to feasibly apply this QI tool in practice must also be considered. While intensive interviews and the standardized walkthrough procedure are achievable within a research context on quality monitoring, such methods are highly time and labor-intensive. This research team is developing and testing methods to more quickly and reliably benchmark standards of practice using this QI tool. In the next phase of this research, we will test a method to benchmark program and individual staff performances on these QIs through live interviews and walkthroughs tailored to directly address these QIs; where possible we will assess the feasibility of concurrently rating performance (e.g., during interviews).

Despite these limitations, the dearth of direct empirical data to support the efficacy or effectiveness of parent-delivered family support services makes such QI tools imperative and timely as the health care system rapidly approaches a different era in health care that emphasizes accessibility, care coordination and accountability of services to improve quality and reduce costs. To the best of our knowledge, this is the first set of QIs for family support that has been developed and piloted. We believe it provides a useful tool to identify factors that may influence the effectiveness of family support services delivered by parents. A companion paper (Olin et al., this issue) supports this notion, demonstrating the association between program performance on these QIs and organizational social context (which has been shown to be related to a number of service outcomes over decades of research). As such, these QIs may have utility in guiding policies and practices in organizations that seek to integrate this service model within children’s mental health care. This paper and others in this special issue represent a first step toward testing the reliability and validity of this quality measure. Further research has the potential to link quality performance to parent and child outcomes, a key step toward establishing the evidence base for parent-delivered family support services. Such efforts are underway by the research team. If better quality family support services are found to be related to parent and child outcomes, future applications could include use of such findings to coach or retrain personnel at the agency and practitioner levels; useful indicators could be adapted as a tool in supervision. As evidence accumulates to support the link between QIs and key outcomes of interest to stakeholders, these QIs may also be useful as part of a credentialing or certification process for organizations that integrate this service model.