Measuring sustainability of a grassroots program in a large integrated health care delivery system: the Warrior to Soul Mate Program

Abstract

Introduction

Veterans experience many physical and psychosocial adjustment problems that challenge personal relationships and social functioning and successful social reintegration. The Warrior to Soul Mate (W2SM) program uses a structured curriculum [i.e., the Practical Application of Intimate Relationships Skills (PAIRS)] to address veterans’ interpersonal needs by teaching participants effective interpersonal skills. Veterans who attended the W2SM program reported lower anxiety levels, improvements in marital alterations and satisfaction, and increased intimacy, cohesion, and affection. Therefore, sustaining the W2SM program can have long-term positive effects for veterans, families, and the greater society. The purpose of this paper is to describe the sustainability of the W2SM program.

Methods

The Model of Community-based Program Sustainability conceptually guided the evaluation. Twenty-three VA hospitals in the U.S. that offer W2SM programs completed a self-report survey to measure sustainability.

Results

The highest scoring sustainability elements were “Demonstrating program results” (M=5.82, SD=1.23), “Staff involvement and integration” (M= 5.79, SD= 1.34), and “Program responsivity” (M=4.39, SD= 1.16); the lowest scoring element was “Strategic funding” (M=2.78, SD=1.75). Statistically significant associations were found between the global middle-range program results and three sustainability elements: leadership competence (r = .472, p = .023), effective collaboration (r = .470, p = .024), and strategic funding (r = .507, p = .014).

Discussion

Efforts to sustain programs should focus on leaders planning for sustainability at the onset of program implementation, collaborators must be involved in program design, implementation and evaluation, and long-term funding sources must be secured to support program operations and continuation.

INTRODUCTION

Since September 11, 2001, more than 2.4 million veterans have returned from deployment to Iraq or Afghanistan.¹ These veterans are faced with multiple physical, psychological, and social problems.^1–4 Specifically, high rates of mental health disorders, poor social functioning, and interpersonal conflicts have been reported.^{4, 5} Military deployment is also associated with poor marital and relationship adjustment and satisfaction, and the lack of intimate personal relationships.⁶ Yet, loved ones play a critical role in offering social support to veterans and facilitating their re-integration. Poor social support post-deployment is also associated with increased PTSD and decreased resilience and psychosocial functioning.⁷

The Warrior to Soulmate program

One mechanism that is available to veterans to address the stress that military service places on marriage relationships is the Warrior to Soul Mate (W2SM) program.⁸ The W2SM program incorporates the structured curriculum of the Practical Application of Intimate Relationship Skills (PAIRS) Essentials program. During the program, trained and certified instructors teach participants effective communication and conflict resolution skills to sustain relationships.^9–11 The PAIRS Essentials curriculum has been used in a variety of formats, including intensive weekend workshops and 60–90 minute sessions over six to ten weeks.^9–11 Studies have shown that participants who attended programs utilizing the PAIRS curriculum reported lower anxiety levels, improvements in marital alterations and satisfaction, and increased intimacy, cohesion and affection.^{12, 13} Support for the wide-scale adoption of the W2SM program by individual Veterans Administration (VA) facilities was demonstrated and facilitated through grants awarded by the VA Office of Patient Centered Care and Cultural Transformation (OPCC&CT) to pilot and expand the W2SM program across VA facilities. As the number of facilities implementing the program grew, a request was made to evaluate the implementation and sustainability of the W2SM program.

The sustainability of the W2SM program

Understanding the sustainability of the W2SM program is important for several reasons. The VA does not usually offer W2SM-like programs to veterans and their families. That is, the W2SM program is not a medical or psychiatric intervention being offered in response to a diagnosis (typical medical model used in healthcare), but rather an innovative, “preventative care, whole health-” intervention that involve the veteran and his/her spouse. The W2SM program received unique funding to offer weekend retreats; funding that is not recurring. Hence, the program could easily be discontinued without the buy-in at local levels to sustain the program and the necessary funding and other resources to support the continuation of the W2SM program.

How to sustain the W2SM program is, however unclear. Understanding the factors that potentially drive its sustainability may allow leaders to take proactive steps to support facilitative factors and reduce or eliminate those factors inhibiting sustainment. Knowledge of the pertinent factors may further serve as a roadmap for the implementation of other psychosocial and clinical interventions in the context of veteran’s care. Leaders and researchers could use such a roadmap to evaluate the pre-implementation context and develop strategies to optimize implementation and maximize long-term sustainability. Without active steps to support program sustainment over time, the benefits derived from implementing the W2SM program will be limited. That is, improvements in veteran’s psychosocial well-being will remain limited while organizational resources, used during implementation, are wasted.

Several studies have examined the implementation and sustainability of healthcare programs ^14–22 Key factors identified across studies included organizational capacity, leadership support, having a program champion, program design and characteristics, program fit and adaptability, the ease of integration of the new program or intervention into existing structures, the need for front-line staff involvement, and staff perception of benefits from program implementation and having a program champion. In a recent systematic review, Stirman and colleagues ²³ categorized an array of influencing factors. These included context (e.g., culture, organizational structure, and policies), processes (e.g., evaluation and alignment of intervention with the setting), the innovation itself (e.g., fit and adaptability), and the capacity to sustain (e.g., funding, resources, and workforce characteristics). Their schema brought some clarity to the large array of factors identified across studies and may serve as an important framework for investigating sustainability factors in future research.

Due to the dearth of studies on the sustainability of the W2SM program, it is unclear what factors are important to the implementation and sustainability of the W2SM program. As part of the request by OPCC&CT for an evaluation of the implementation of the W2SM program, ²⁴ we conducted an evaluation of specific factors and their association with the sustainability of the W2SM program. The study drew on one of the few models of program sustainability available in the literature, namely the Mancini and Marek Model of Community-based Program Sustainability²⁵ to identify important factors associated with program sustainability.

CONCEPTUAL FRAMEWORK

Mancini and Marek’s Model of Community-based Program Sustainability was used to conceptually guide the evaluation of the sustainability of the W2SM program.²⁵ Mancini and Marek propose that six sustainability elements are important to achieve both middle-range program outcomes and the long-term outcome of sustainability.²⁵ Without achieving mid-range program outcomes, the long-term sustainability of projects is unlikely. The six sustainability elements include leadership competence, effective collaboration, demonstrating program results, strategic funding, staff involvement and integration, and program responsivity. Leadership competence is reflected in the presence of a well-developed program’s vision and objectives, program assessments and evaluations, adapted programs as needed, secure funding, and training, support and supervision of staff. The importance of leaders’ competence and understanding in the sustainability of innovations is well-known.^{26, 27} Effective collaboration refers to the involvement of stakeholders who actively support the goals of the program and have clearly identified responsibilities. Programs are more likely to be accepted when stakeholders’ input is sought and their support for the program is gained.^{15, 28} Demonstrating program results include program evaluation and activities to report and share the results with stakeholders. Sharing program results increases the visibility of programs, the potential for buy-in from organizational members, staff engagement and could guide future educational efforts.²⁹ Strategic funding is defined as having the necessary plans and resources in place to support the continuation of the program. The availability of resources and funding are imperative for sustainability and organizations must be intentional in securing the necessary internal funding to support the sustainability of programs.^{28, 30} Staff involvement and integration refers to the involvement of committed, qualified staff when the program was designed, implemented, and evaluated as well as during decision-making. The inclusion of organizational members during the adoption and implementation of an innovation has shown to facilitate sustainability.^{29, 31, 32} The importance of program responsivity—the adaptability of the program to the needs of a community—to program sustainability has been well demonstrated ^{16, 33} Middle-range program results are intermediate to the program becoming sustained. These results included the perceived ability of the program to meet the needs of the individuals who were targeted by implementing the program, leaders planning for program sustainability, and staff members having confidence in the survival of the program. Mancini and Marek proposed these results are objectives of a program and that these results are closely associated with ultimate long-term sustainability.²⁵ Sustainability of programs is also unlikely unless they have transitioned from pilot programs to permanent status. Therefore, program permanent status was included in the model as an additional middle-range program result.

METHODS

Setting and Sample

The initial adoption of the PAIRS marriage enrichment workshop was largely due to the work of one chaplain from a single VA who observed the stress that combat experiences placed on veterans and their relationships. After successfully pilot testing the use of the PAIRS model within his VA facility, and with grant support from OPCC&CT in Washington (DC), a second chaplain adopted the model for conducting PAIRS events (e.g., weekend retreats), developed a user’s guide to implement weekends, and managed the distribution of grant dollars. Subsequent national adoption of the W2SM by individual VA facilities was facilitated through grants awarded by the OPCC&CT. Our evaluation of sustainability focused on the VA facilities receiving OPCC&CT innovation grant funding in 2013 (n=23).

In 2013, of the 27 facilities that originally received OPCC&CT funding, 23 facilities held W2SM events. These medical centers were located in 14 of the 23 administrative regions of the VA (Veterans Integrated Service Networks), located across the United States. A total of 67 W2SM programs were held, serving 826 couples and 12 individuals who attended by themselves. The majority of the W2SM retreats were held in hotels or inns (41, 61%) while 15(22%) were held in other locations such as retreat centers. Very few retreats were held at VA medical centers (6, 9%). The 23 sites reported a total of 71 personnel being involved with the W2SM program. Of these, 36 (43%) were from the chaplain service, 19 (23%) were from the social work service, and 16 (19%) were from the mental health service.

All of the VA facilities who received funding were targeted for participation in the evaluation (n=23). The OPCC&CT funded a collaborative evaluation of the sustainability of the program to include barriers and facilitators of potential sustainability for the W2SM program. This evaluation was conducted by a multi-disciplinary team from one of the VA Health Services Research & Development Centers of Innovation (COIN) and VA Mental Health and Chaplaincy program, in collaboration with representatives from the OPCC&CT, VA National Chaplain Center, national leadership of the W2SM program, and one of the VA Geriatric Research Education and Clinical Centers (GRECC). Detailed information on the evaluation of the W2SM program is described elsewhere.²⁴

The Innovative Intervention

The PAIRS Essentials program, developed by the PAIRS Foundation, has a set curriculum that is offered by PAIRS certified instructors as a weekend retreat.⁸ Within the VA, instructors included chaplains, social workers, nurses, and other mental health providers. The program is designed to increase participants’ self-knowledge and to develop their ability to maintain enjoyable intimate relationships.^{8, 11, 13} Outcomes of the program include participant reports of lower levels of anxiety and conflict, and increased intimacy and affection and improved relationships.^{9, 10, 12, 13}

Data Collection and Measurement

Site leaders at each facility completed two self-report surveys. The first survey collected information about individual W2SM events (e.g., who was served, type and location of events). The second survey collected information at the programmatic level for the 23 participating medical centers. Drafts of the surveys were reviewed by three members of the W2SM National Consultative Council to determine if they interpreted questions in the way intended by the evaluation team. The surveys were conducted via the VA’s internal Web system. W2SM contacts were emailed the survey link by the Chairperson of the W2SM National Consultative Council with follow-up emails sent by the evaluation team approximately every two weeks until all surveys were returned.

The programmatic survey included the Program Sustainability Index (PSI), used to measure six sustainability elements. The PSI consists of 29 items and five subscales, namely leadership competence (5 items; α= 0.899), effective collaboration (10 items; α= 0.934) demonstrating program results (4 items; α=0.916), strategic funding (3 items; α=0.803), staff involvement and integration (4 items; α=0.756) , and program responsivity (3 items; α=0.401). A PSI composite score was also generated as a mean score of the 29 PSI items (α = .881). Responses were measured on a 5-point Likert scale (1=not at all to 5= Very much).

The 29-item PSI is the result of a confirmatory factor analysis (principal axis analyses with oblique rotations) (factor loadings ≥ 0.30 was deemed acceptable) on the initially proposed 53-item PSI. Examples of questions included in the survey are: Leaders developed and followed a realistic PAIRS project plan” (leadership competence); “Local decision makers are involved as PAIRS program collaborators” (effective collaboration); “Evaluation plans are developed prior to implementing the PAIRS program” (demonstrating program results); “Current funding is sufficient for the PAIRS program’s operations” (strategic funding); “PAIRS’ staff are involved in PAIRS program design” (staff involvement and integration); and, “PAIRS programs are eliminated when they do not meet community needs” (program responsivity). The validity and reliability of the PSI has been established previously.²⁵

Four questions were used to measure middle-range program results: (1) The PAIRS program meet the needs of the participants.” (2) “I am confident that the PAIRS program will still be active in five years.” (3) “We have a plan in place to continue PAIRS after FY 2013/use of OPCC&CT funding”, and (4)” The PAIRS program has achieved permanent status.” Items in the PSI and responses to the middle-range program result questions were scored on a 7-point Likert scale (1=strongly disagree; 7= strongly agree).

Data Analysis

The unit of analysis was the facility/ site where the W2SM program was implemented. Quantitative results were evaluated using the Statistical Package for the Social Sciences (SPSS) (Version 19.0.2). Counts were used to summarize the program and participant numbers. Mean (M) and standard deviation (SD) were used to summarize the normally distributed PSI and middle-range program scores. One of the middle-range scores (meets needs) had a skewed distribution, therefore the median and interquartile range (IQR) summarized those scores. Correlations among the PSI elements and correlations of those elements with the middle-range program scores (with the exception of the “program meets needs scores”) were conducted using Pearson correlations. Spearman Rho coefficients were used to assess those correlations. A Cronbach’s alpha of 0.05 was used for statistical significance.

Ethical Considerations

This project was certified as a non-research operations activity by the Director Office of Patient Centered Care and Cultural Transformation (OPCC&CT), Veterans Health Administration, in accordance with VHA Handbook (regulation) 1058.05.

RESULTS

Sample Characteristics

All of the 23 sites that held W2SM programs completed the study survey (100% response rate). Summaries of the number of retreats held and participants in those retreats are contained in Table 1.

Table 1.

Warrior to Soul Mate Program Implementation in Fiscal Year 2013 across 23 sites.

	N
Retreats held in FY 2013	67
Participants
Couples	826
Singles	12
Veterans	847
W2SM Personnel Service
Chaplain	36
Mental Health	19
Social Work	16

Program Sustainability and Middle-Range Program Results

Summaries of the Program Sustainability Index (PSI) and Middle-range scores for the 23 sites are summarized in Table 2. The PSI scores ranged from 2.8 − 5.8 (Mean = 4.4, SD = 0.76) (see Table 2). Of the program sustainability elements measured by the PSI, the highest score was reported for “Demonstrating Program Results” (Mean = 5.8, SD =1.23), followed by “Staff Involvement and Integration” (Mean = 5.8, SD = 1.34). The lowest scoring sustainability element was “Strategic Funding” (Mean = 2.8, SD = 1.23), followed by “Leadership Competence” (Mean = 3.8, SD = 1.67). (See Table 2)

Table 2.

Program Sustainability Index (PSI) Scores and Middle-range Program Results (N=23 sites).

	Mean [SD]
PSI Construct
Leadership Competence	3.8 [1.67]
Effective Collaboration	4.0 [1.64]
Demonstrating Program Results	5.8 [1.23]
Strategic Funding	2.8 [1.75]
Staff Involvement and Integration	5.8 [1.34]
Program Responsivity	4.4 [1.16]
Total PSI Score	4.4[0.76]
Middle-range program results
Total Middle-range Program Result Score	4.5 [1.52]
Confidence that the program will still be active in five years	4.1 [2.13]
A plan is in place to continue W2SM after FISCAL YEAR2013/use of OPCC&CT funding	4.4 [2.17]
The W2SM program has achieved permanent status (i.e., it is one of the services now offered permanently) in the medical center	3.1 [1.92]
The W2SM program meets the needs of participants	7.0 [6.0–7.0]a

Note.

^a= Due to skewness, Median, and 25^th and 75^th Interquartile Range are reported.

Overall middle-range program result scores ranged from 1.75 − 7.0 (Mean = 4.5, SD = 1.52). Of the individual middle-range program results, the highest score was reported for the program meeting the needs of participants (Mean = 6.4, Median=7.0, IQR = 6.0–7.0), whereas the lowest score was reported for the program achieving permanent status (Mean = 3.1, SD = 1.92) (see Table 2).

Associations among the PSI scores indicated statistically significant positive correlations between the sustainability elements of “Leadership Competence” and “Effective Collaboration” (r =.43, p =.038), as well as between “Demonstrating Program Results” and “Staff Involvement and Integration” (r =.75, p < .001). No other statistically significant associations among the elements were observed (see Table 3). (see Table 3)

Table 3.

Correlations among Program Sustainability elements. (N=23).

	Leadership Competence	Effective Collaboration	Demonstrating Program Results	Strategic Funding	Staff Involvement and Integration	Program Responsivity
Leadership Competence	--	.43 (.038)	.16 (.454)	.20 (.353)	.12 (.597)	.25 (.258)
Effective Collaboration	--	--	.05 (.825)	.37 (.082)	.09 (.688)	−.05 (.834)
Demonstrating Program Results	--	--	--	−.29 (.182)	.75 (<.001)	−.01 (.952)
Strategic Funding	---	---	---	--	−.17 (.443)	−.26 (.230)
Staff Involvement and Integration	--	--	--	--	--	−.09 (.685)
Program Responsivity	--	--	--	--	--	--

Note: Values in the cells are r (p-value).

Summaries of the associations of the PSI scores with the middle-range program scores are shown in Table 4. The associations between “Leadership competence” and “Strategic funding” and the global middle-range program result score were also statistically significant (Table 4). A statistically significant association was observed between “Effective collaboration” and only one of the middle-range program results, (“Active in five years” [r =.48, p=.022]). However, this variable showed a statistically significant association with the global middle-range program result score (Table 4). The sustainability element with the strongest association with middle-range program results was “Strategic funding” (r = .51, p=.014). (See Table 4)

Table 4.

Correlations of Independent Variables with individual Middle-Range Program Results. (N=23).

Characteristic	Meet participant needsa	Active in five yearsb	Plan for continuance in placeb	Achieved permanent statusb	Middle-Range Program Resultsc
Leadership Competence	−.03 [.911]	.52 [.011]	.44 [.035]	.48 [.021]	.47 [.023]
Effective Collaboration	.14 [.526]	.48 [.022]	.35 [.105]	.47 [.023]	.47 [.024]
Demonstrating Program Results	−.13 [.568]	.07 [.760]	−.06 [.778]	−.01 [.969]	−.03 [.882]
Strategic Funding	.05 [.820]	.51 [.013]	.49 [.018]	.46 [.026]	.51 [.014]
Staff Involvement and Integration	−.20 [.358]	.19 [.395]	.19 [.378]	−.00 [.992]	.10 [.667]
Program Responsivity	−.20 [.352]	.23 [.290]	.22 [.318]	.16 [.446]	.20 [.357]

Note: Values in the cells are r [p-value].

^a=Spearman Rho coefficients due to skewness

^b=Pearson Correlation coefficients

^c=composite score

Statistically significantly positive associations were, however, observed of the sustainability elements “Leadership competence” and “Strategic funding” with the other three middle-range program outcomes of “ Program active in five years,” “Plan for program continuance,” and “Program achieved permanent status” (Table 4).

DISCUSSION

In 2013, 67 W2SM retreats were offered by 23 VA hospitals across the U.S. and 1,664 people, with 826 couples and 847 veterans attending. Seventy-one staff members from chaplaincy, mental health, and social work conducted the weekend retreats.

This study found variability in the presence of the six sustainability elements across the 23 sites. Although there is some room for improvement, the W2SM program has established processes to evaluate the program (i.e., demonstrate program results), which includes involving front-line staff in the design, implementation, evaluation, and decision making surrounding the program (i.e., staff involvement and integration). Staff has also adapted the program to meet the changing needs of the community (i.e., program responsivity). These three sustainability elements achieved the highest mean scores of all the sustainability elements, however, a non-statistically significant association was observed between these elements and the global middle-range program result score. Therefore, in contrast to other studies ^{30, 31, 33} the findings of this study indicate that evaluation activities, including involvement by front-line staff, and adapting the program were not important for W2SM program sustainability.

This study also found that the sustainability elements with the lowest mean scores—leadership competence, effective collaboration, and strategic funding—were significantly and most strongly associated with W2SM sustainability. Other studies have also similarly recognized the importance of leadership, collaboration, and funding to sustainability.^{28, 31} Of all the sustainability elements, “Strategic funding” achieved the lowest mean score of all the sustainability elements, yet had the strongest association with middle-range program results.

The findings of this study indicate that a focus on contextual factors does not guarantee program sustainability. A variety of contextual factors were present in facilities that offered the W2SM program, yet, only some factors were associated with sustainability. Research indicates that the operationalization of sustainability factors (the “how”) in organizations distinguished high versus low sustainability organizations^{22, 34} Furthermore, implementation strategies are now recognized as important in implementation and sustainability.^35–37 Therefore, although factors were present in the W2SM program, how they were implemented or operationalized may have inhibited their overall role in program sustainment. Future evaluations of the W2SM program and its sustainability should include an assessment of the implementation strategies facilities used and their impact on program sustainability

This study did not investigate the role of the characteristics of the W2SM program on sustainability. However, Rogers’ Diffusion of Innovation theory³⁸ recognizes the importance of program characteristics (i.e., relative advantage, complexity, compatibility, and trialability) on adoption and implementation. Because the W2SM program was unique (i.e., not a medical or psychiatric intervention), facilities may have perceived the W2SM program as beneficial (relative advantage). However, its complexity (e.g., multidisciplinary approach and extensive use of resources like staffing and time) may have limited the implementation of all program components. Future research should investigate the role of program characteristics in the long-term sustainability of the W2SM program. Studies should also identify the core components of the W2SM program that are required to achieve positive patient outcomes in order to reduce program complexity and resource demands. This is particularly important given the demonstrated strong association of organizational capacity with the sustainability of the W2SM program.

Two main limitations of this evaluation are that the evaluation is based on cross-sectional data and a self-report survey. Therefore, causal relationships between variables cannot be established and response bias may be present. Future researchers should conduct longitudinal studies and include a variety of data collection techniques.

PRACTICAL IMPLICATIONS

The findings of this study have practice implications. Organizational leaders and program directors can take several steps to increase the potential for the sustainability of programs. Organizational leaders and program directors should focus their efforts on establishing and articulating the mission and vision of programs and start planning for sustainability at the onset of program implementation. Such planning should include the identification of key stakeholders and collaborators and facilitating their involvement in implementation activities. Program implementation plans should be realistic and strategies to achieve the continuation of programs should be developed. In the case of W2SM programs, leaders should articulate the goals of the program and develop strategies for the continuation of programs as part of the initial planning phase. Effective collaboration during the implementation processes should be established and the necessary stakeholders brought into the design, implementation, and evaluation of programs such as the W2SM program. Roles and responsibilities of staff members involved in the development and execution of the program should be well-defined and successful strategies shared by all staff involved. Most importantly, strategic planning related to funding is imperative to continue to facilitate the long-term sustainability of programs such as the W2SM program. This would be particularly important in the case of programs supported through grant funding.

In sum, when implementing the W2SM program, more emphasis should be placed on allocating and securing resources to support the program, leaders focusing on program related initiatives and quality control, and establishing partnerships with stakeholders who support the W2SM program. Careful consideration is needed to how facilities operationalized various sustainability elements, the strategies they used to implement the W2SM program, and on program characteristics.