Table 2.

Case study example using the Implementation Planning Assessment Tool ¹

Implementation Planning Assessment Tool for the Veterans Health Administration (VHA) Cooperative Studies Program (CSP) and other clinical trials
Guidance: As a team, review and document responses to the overarching questions. The intention is for this tool to be completed as an iterative process and the teams and individuals can and should refer to the tool at different points in time throughout the trial.
Phase 1. Planning, Framing, and Aligning Interested Parties: How do CSP trial programs identify and align interested parties? Planning, framing, and aligning interested parties help inform the design of the intervention to be implemented (e.g., design-for implementation, user-centered design). Clinical trial programs often include national interested parties upfront on their Executive Committees, such as national program office leads (e.g., Pharmacy Benefits Management, National Pathology & Laboratory Medicine, Patient Care Services, Clinical Services program offices). However, these programs should also consider identifying and collecting input from potential end-users at the regional (e.g., Veterans Integrated Service Network (VISN), Chief Medical officers (CMOs)) and local levels (e.g., facility Chiefs of Staff and Service Line Chiefs), and Veteran or patient users throughout the trial process. Involving interested parties at multiple levels will also help enhance equity and diversity in implementation planning and garner buy-in, at the local clinic (e.g., frontline provider, facility service line) and regional managerial levels (e.g., VISN Director, CMO).
1. What is the intervention/treatment and what are its core elements that are hypothesized to achieve its desired effect on health? Intervention: Individual Placement and Support (IPS) is the evidence-based model of supported employment and is highly effective in helping Veteran patients who are diagnosed with post-traumatic stress disorder (PTSD) obtain and sustain employment in competitive jobs. Core Elements: An IPS Specialist provides personalized employment services that follow these core elements: (a) zero exclusion for eligibility, (b) rapid job search; (c) systematic job development in a diversity of jobs; (d) competitive employment rather than set-aside or transient jobs; (e) client choice for well-matched and meaningful jobs; (f) integration of IPS within the clinical treatment team ensuring shared decision making between providers and Veteran clients; (g) personalized benefits counseling; and (h) individualized support during follow-up for as long as needed even after a competitive job is obtained. Desired Effect on Health: To acquire meaningful sustained competitive employment in a well-matched job to help a person with a disability overcome personal, professional, and interpersonal difficulties. This is particularly important for someone living with PTSD, since real-world experiences through competitive work helps therapeutically break through the patient’s behavioral isolation/avoidance, cognitive distortions, and emotional reactivity. Employment is often the “hook” to motivate the patient to engage in treatment programs to better their chances of recovery and further success.
2. What clinical issue or public health problem is the intervention trying to solve? Clinical Issue: Unemployment or occupational difficulty that have broad negative impacts on quality of life, physical and mental health outcomes (including increasing the risk of suicide and/or addictions), and societal economic burden. Intervention: The IPS intervention aims to positively impact the likelihood of obtaining and maintaining employment, which is a clinically relevant functional outcome for a disabling condition such as PTSD. VA Cooperative Study #589 Veterans Individual Placement and Support Toward Achieving Recovery (VIP-STAR) targeted steady competitive employment for Veteran patients, as its primary outcome. Rather than selecting a PTSD symptom outcome, the primary outcome was steady employment, which is a functional outcome that is more relevant to the quality of life and recovery of the Veteran with PTSD. Functional recovery in work sets forth a trajectory which improves psychological outcomes and reduces negative health outcomes over time.
3. Create a list of the key interested parties involved in the intervention/treatment (including those with high interest in the treatment or intervention, as well as key influencers in its adoption and sustainment in routine practice over time, and those who may provide insight on equity and diversity considerations, for example, patients, clinical and non-clinical staff, local and regional managers, national program office leads, policymakers, and recruitment of diverse patients (race/ethnicity, gender, LGBTQ+, disability) in trials, for example, the VA Women’s Enhanced Recruitment Process (WERP) 2 ):   • Unemployed Veterans living with a diagnosis of PTSD who have an interest in gaining competitive employment   • Local facility PTSD care providers and mental health leadership   • Community employers and workforce development organizations   • VA’s Office of Mental Health and Suicide Prevention (OMHSP)   • National Mental Health Director for Psychosocial Rehabilitation and Recovery Services at VHA10NC5ACTION@va.gov   • VA National Center for PTSD   • Northeast Program Evaluation Center (NEPEC) in OMHSP   • Veterans Integrated Service Network Directors and Facility Directors   • Veterans Integrated Service Network Chief Mental Health Officers and Vocational Rehabilitation Service Program Managers, Staff, and IPS Specialists   • Veterans Integrated Service Network Supported Employment Mentor-Trainers   • VA Medical Facility Local Recovery Coordinators   • IPS Learning Community (non-VA IPS practice leaders in the field) See VHA DIRECTIVE 1163 August 13, 2019, for definitions and roles for these interested parties .
4. What staff and resources do you have to support preliminary implementation planning work and delivery of the intervention/treatment?   a. For planning of the study including treatment or intervention/treatment delivery.   b. For capturing data on the process by which the treatment is being implemented, such as provider delivery and fidelity to the treatment (e.g., checklists of treatment core elements completed and delivered to each patient), data capture of the treatment use by existing sources for future surveillance (e.g., Electronic Health Record (EHR), diagnostic, lab, treatment codes), surveys, interview guides, software, etc. for capturing interested party perspectives of the treatment (patient, provider, manager, leaders). Consider including Full-Time Equivalent (FTE) for these tasks.   c. Consider staff FTE, protected time, supplies, and equity and diversity of facilities or staff implementing the intervention (audio recorder, headset, software, training). The project did not have existing IPS specialists in place at the local site who could serve a full caseload of Veterans with a diagnosis of PTSD. Thus, CSP #589 funded one or two IPS specialists at each local study site. The CSP Executive Committee’s IPS experts gave input into the selection of the new IPS specialists and at times, the onboarding of the new IPS specialist(s) was held up by Human Resources due to the IPS candidate not having a specific educational degree in vocational rehabilitation counseling. This advanced degree does not necessarily translate into a person with the training, skill set, or understanding of the IPS practice. Advocacy from the local site investigator and CSP Executive Committee IPS experts were needed to facilitate the onboarding of suitable candidates to fill the IPS specialist positions. In addition, the teams had to work together when there was turn-over in the IPS specialist positions or if an IPS incumbent had performance deficits. In addition, training, ongoing technical assistance, and fidelity monitoring for the local IPS Specialists were carried out by CSP-funded IPS Trainers and an IPS Fidelity Monitor as part of the initial kickoff visit and subsequent on-site visits at the participating sites. This ensured the fidelity of the IPS model was maintained while any barriers of model implementation at the local facility were addressed. For example, bringing together the clinical and research teams at each participating facility ensured first and foremost the research project kept the Veteran participant’s well-being and welfare at its core; no issue went unnoticed irrespective of whether it came from the research team or the clinical team. This in turn, ensured the data processes followed in the research protocol were complete and in sync with clinically collected information, since both teams were aware of the Veteran’s participation in the study and whether they were assigned to IPS or the control intervention. The control group intervention (Transitional Work Program) was provided by each local facility’s existing vocational rehabilitation services. The study team had to work with some sites to ensure that the control group honored the time-limited nature of transitional work assignments (i.e., the set-aside job needed to be kept within a national standard of approximately 90-day length rather than a protracted 6 months or more that some sites had evolved into over many years prior to the study implementation). The CSP study also funded one FTE Clinical Research Coordinator at each site to manage regulatory processes, informed consent, baseline and follow-up assessments, data collection, data submission to CSP, and data queries. Each site had a designated local site investigator (not funded by CSP) who was a provider embedded in either the PTSD team or the vocational rehabilitation team.
5. What level of facility, regional, and national leadership support is there for the intervention (describe from high to low)?   a. For facility leadership explain the type of support that leadership is providing for the intervention/treatment (e.g., FTE for clinical trial, prioritizing and spreading support for the intervention or treatment trial recruitment, etc.).   Note: This step includes assessing the current leadership support at involved sites (facilities) and using strategies to increase that support as necessary 3 Qualitative interviews, focus groups, or telephone conversations are methods often used to obtain feedback from leadership. 3
National: National leadership provided support in the planning phase. VA CSP established a planning committee that included some national interested parties during the design phase. After the study was approved for funding, the VA CSP #589 Executive Committee was formed and included representatives from NEPEC, local study sites, and other IPS experts in VA and non-VA settings. A national consultant was funded by Quality Enhancement Research Initiative (QUERI) to conduct qualitative interviews with the IPS Specialists and local site interested parties during the later stages of the project. National and Local Site Collaboration: The CSP project also funded an IPS expert who conducted semi-annual IPS fidelity monitoring visits at the local sites. His work at the local site level focused on discussions with the study teams and others at the facility to ensure the IPS intervention was successfully launched and sustained. A fidelity monitoring visit ended with a debrief session with the facility Director, front office Quad members, Service Line Directors, and vocational rehabilitation program managers. These debrief sessions with the facility leadership often included presentations of actual case examples of veteran participants and their lived experiences of prior unemployment and the success of IPS in restoring their treatment goals. Issues were also brought back to the Study Executive Committee for discussion and action as necessary. Local Site Support: The local facility leadership provided 1) the protected time for the local site investigators who were responsible for the protocol-related care and coordination of the research participants, 2) the PTSD treatment team providers who were responsible for the clinical treatment of the participants, 3) the vocational rehabilitation specialists who provided the control intervention, and 4) authority to work through barriers of study-related processes and IPS implementation when needed (e.g., human resources, information technology, research services, government car motor pool, and space).
6. Have frontline users (clinical and non-clinical staff) provided input on the design and deployment of the intervention? Note: Qualitative interviews, focus groups, or conversations are methods often used to obtain feedback from frontline users 3 Clinical and non-clinical input was provided in the planning stages of the study. The CSP Planning Committee was a diverse group of clinicians, methodologists, and subject matter experts. Focus groups and interviews did not take place during the planning process; however, these took place in the later stages of the study. Frontline clinical providers (users) were members of the CSP #589 Executive Committee and met quarterly to discuss implementation and enrollment issues. Annually, the local site investigators and IPS specialists gathered in-person to go over the study status and discuss IPS implementation. To allow for in-depth discussions at the annual meetings, breakout meetings were held for investigators and study coordinators to discuss study protocol issues and for IPS specialists to discuss implementation practices). More feedback from local site frontline users should have been solicited outside of the IPS Fidelity Monitoring visits so that users could have felt more open to share concerns and express difficulties.
7. Which determinants framework 3 will be used to identify and describe contextual factors that could influence the implementation process and quality of intervention protocol delivery?   a. The framework informs data collection and analysis to identify barriers and facilitators to practice uptake.   Note: Examples of determinants frameworks include Theoretical Domains Framework (TDF), Consolidated Framework for Implementation Research (CFIR) [19], and the integrated-Promoting Action on Research Implementation in Health Services (i-PARIHS) [20]. None of the above. As a framework, a published structured IPS Fidelity Scale was used to evaluate the IPS implementation, quality, and facilitators/barriers to implementation. Fidelity monitoring visits were conducted semi-annually and involved in-depth interviews with leadership, staff, treatment providers, Veterans, and employers, in addition to chart reviews of IPS recipients and observations of team meetings and IPS community interactions with Veterans and employers.
8. Who are the multi-level interested parties of the intervention (e.g., those who have an interest in using the treatment or intervention if proven effective, those who may provide insight on equity and diversity considerations of individuals and/or healthcare settings, and/or those who have influence on the policies that would foster sustainment of the treatment in routine practice if proven effective)?   a. Describe how interested parties were identified and how their perspectives will be assessed over the course of the trial at each participating site (e.g., qualitative or quantitative methods).   b. Identify the local and regional interested parties involved in the intervention.   c. Map out who needs to take different steps (role) and when in the process for the intervention to be successfully implemented.   d. Identify Veteran or patient interested parties. Interested Party Input: The National Mental Health Director for Psychosocial Rehabilitation and Recovery Services (at this time called Director for Compensated Work Therapy) and a representative from the Northeast Program Evaluation Center were interested parties involved in the planning for the intervention. The leadership at the National Center for PTSD was kept informed as to the planning of the study and gave informal input. In hindsight, the project would have benefited from more formal input from the VHA’s Office of Mental Health and Suicide Prevention (OMHSP) regarding sustained post-study implementation of IPS. Site Selection: An email was sent from CSPCC to the research offices and facility directors at all VA Medical Centers announcing the study and asking for interested medical centers to fill out a survey. Based on the results of this survey, size and demographics of the PTSD population, record of success of prior IPS implementation for other populations, track record of research success, presence of a CSP NODE, and local site investigators’ qualifications, 12 sites were identified to serve as the CSP sites. Post-selection: Facility leadership, the PTSD care team, and the research team worked together at each participating VA Medical Center to share and discuss the implementation of the study and the plan of care for Veterans enrolled in the study.
9. How have multi-level interested parties who have an interest in the intervention/treatment provided input? Interested parties should include Veteran or patient perspectives as well. For all interested parties, elicit feedback about equity, diversity, inclusion, barriers, facilitators, and satisfaction with the intervention/treatment.   a. Consider local, regional, national interested parties.   Note: Qualitative interviews, focus groups, or conversations, advisory calls, are methods often used to obtain feedback from key interested parties. 3 The proposal was reviewed by the Human Rights Committee which has Veteran representation. This was the only instance where Veteran input was provided. The investigators had prior experience with IPS implementation and/or with PTSD populations. Qualitative interviews with consumer-interested parties were conducted to evaluate the barriers, and facilitators of IPS implementation were collected over the course of the study (Pogoda QUERI project). At the end of the study, participants completed a satisfaction survey.
10. Which national program officers and operational partners could facilitate pathways for future spread or implementation?   • VA’s Office of Mental Health and Suicide Prevention (OMHSP)   • VA Psychosocial Rehabilitation and Recovery Services in OMHSP   • Northeast Program Evaluation Center (NEPEC) in OMHSP   • VA National Center for PTSD   • Veterans Integrated Service Network Directors and Facility Directors   • Veterans Integrated Service Network Chief Mental Health Officers   • VA QUERI and/or VA Diffusion of Excellence   • VA Office of the Secretary
11. Has your implementation plan included each of the following important components?   a. The rationale for the selection of appropriate implementation science theories or frameworks. Yes   b. Methods for quantitative (e.g., intervention uptake based on existing EHR data), qualitative, or mixed methods data collection and analyses. If using mixed methods, how will the methods be integrated? Yes   c. Quantitative sampling plan for control and comparison groups and data codes and fields to capture use in the EHR. CSP implementation leads should coordinate with CSP coordinating centers to determine what data is already being collected to avoid duplication. If not applicable, describe the rationale. N/A; this study preceded the CSP Implementation initiative. Data required for the fidelity monitoring were collected as part of the protocol.   d. Qualitative sampling plan of patient and clinical interested parties (e.g., purposeful criterion, stratified, snowball strategies). If not applicable, describe the rationale. No   e. Specification of the types (quantitative or qualitative) or sources of data (e.g., primary or secondary) to be used, data accessibility, aggregate and subgroup analyses, and provisions for ensuring data quality and adherence to the study protocol. This was described in the proposal; monitoring of data and adherence to protocol was carried out through the CSP processes as specified by the study protocol.
12. What are the preliminary plans for the interventions’ sustainment, once the trial ends, if the intervention/treatment is found effective (see also Phase 3)? The plan should take into consideration any administrative or policy changes needed at the national and regional levels (e.g., formularies, labs, EHR fields, national directives, or other services policies), time, tools, and training required by clinicians at the frontline to deliver the intervention and where (e.g., primary care, specialty care clinics, Community-Based Outpatient Clinics, etc.) and Veteran time required (e.g., visits, required lab tests, medications, etc.)   a. The plan should also take into consideration how pragmatic the trial is (e.g., factors that could impact use of the intervention in real-world settings such as cost, intervention deliverers, intervention recipients). No such plans were defined a priori. At semi-annual fidelity debrief sessions, significant efforts were made to present the case for effectiveness of IPS to the sites’ leadership and encourage the sites to adopt the IPS model for their PTSD population after the study ended. Resource management priorities were often raised as a core challenge by the facility leadership. The leadership valued the IPS model, but resource constraints made it impossible to hire the IPS specialist post-study. The plan did not take into consideration that a major policy change would be needed at the national level (i.e., VHA Directive language and IPS service provision for a new population of Veterans served). Nor did it accurately gauge the tenacity of the treatment-as-usual vocational services that possibly will require reallocation of resources and/or re-training of existing staff in the transition from non-evidence-based to evidence-based service delivery.
Phase 2. Implementation Process Data Collection: How will the implementation process be studied, measured, and assessed? The Implementation Process Data Collection Phase involves ascertainment of factors affecting the use of the CSP intervention or treatment at the routine practice level, notably through information on provider and patient perspectives and acceptance, implementation and intervention costs and organizational factors, and where relevant fidelity to the implementation of the intervention or treatment. This phase also involves enacting an implementation assessment plan and should include equity and diversity considerations throughout.
1. Who is part of your assessment team? Describe the amount and type of time set-aside for the assessment team, implementation lead, or others (e.g., FTE, protected time, donated). The uptake of the intervention was done in an organic way, via further discussions with the participating site investigators. There was significant activity by the Study Chair through discussions/communication with other interested parties that aimed at implementation of the intervention, given its effectiveness and impact on outcomes. Most of this work for post-study implementation was done under the study leadership (i.e., CSP Coordinating Center (CC) and Executive Committee) on donated effort.
2. Have you finalized your assessment plan (#11 in planning) to address the following?   a. The rationale for the selection of appropriate implementation science theories or frameworks.   b. Methods for quantitative, qualitative, or mixed methods data collections and analyses. If using mixed methods, how will the methods be integrated?.   c. Quantitative sampling plan for control and comparison groups for describing the implementation process and intervention uptake and use. If not applicable, describe the rationale.   d. Qualitative sampling plan of patient, local-, regional- and national-level leadership, and clinical interested parties, including specific techniques and measures. If not applicable, describe the rationale.   e. Specification of the kinds or sources of data to be used, data accessibility, aggregate and subgroup analyses, and provisions for ensuring data quality and adherence to the study protocol. N/A (retrospective use of checklist)
3. Which implementation strategies did you select to help attain successful implementation of the intervention?   a. For efficacy trials, define your implementation strategies and their goals.   b. For effectiveness trials, is the trial paying for providers to deliver the treatment/ intervention to the patients or relying on existing providers? How do you intend to measure: 1) fidelity to the intervention or treatment, 2) the uptake or use of the treatment (e.g., patient use or “dose” and 3) what will training, and competencies look like for existing providers once the study ends and the intervention is shown to be effective (e.g., for providers to take up the effective intervention, what level of expertise and training is optimal and what will the manual contain on how they deliver the treatment?) 4) identify the costs of training, 5) consider interested party buy-in for training   c. Document data about your implementation strategy enactment that could help future sites during scale-up and spread if the intervention is found effective   d. To what extent have interested parties provided input into the selection of the implementation strategies?   e. Describe the dose of implementation strategies [16].   Note: Selection of implementation strategies will be based on the key barriers and key facilitators identified during pre-implementation. A structured study-wide process to help attain broad implementation (e.g., implementation at all study sites once the clinical trial ends) was not designed prior to study launch for this intervention. This deficit in planning for implementation likely negatively impacted the sustainment of IPS at the 12 local sites. A key barrier to post-study implementation was lack of funding for the local site IPS specialists designated to serve the PTSD patient population. Precedent existed for VA to support an enterprise-wide rollout of IPS for Veterans with psychotic disorders (2005). The investigators assumed that a similar funding stream would materialize after the evidence emerged for the PTSD population. Leadership changes at the national level led to an attrition in resources for IPS sustainment which resulted in diminished focus on expanding services and/or ongoing quality monitoring across the board. Had the checklist been used prospectively, cost planning would have occurred that likely would have enabled planning to circumvent this barrier. A key facilitator to sites that did later implement IPS (which were not study sites) was a new federal funding opportunity for IPS implementation (pay-for-success social impact bond) involving VA and non-federal partners providing financial support and VA medical centers willing to expand IPS service under the new funding stream.
4. What adaptations or resources are needed for to fit local contexts? Please note for efficacy trials, adaptations may be less relevant or applicable.   a. Describe the core components of the clinical intervention or treatment that are believed to be the causal mechanisms of therapeutic change and are responsible for achieving the intervention or treatment’s desired effects (those facets of the intervention that cannot be adapted or changed).   b. Explain how the intervention can be adapted without compromising fidelity.   c. Consider how adaptations can address or increase equitable implementation of the innovation IPS core principles that are believed to be the mechanisms of therapeutic change and that cannot be adapted or changed: (a) zero exclusion for eligibility; (b) rapid job search; (c) systematic job development in a diversity of jobs; (d) competitive employment rather than set-aside or transient jobs; (e) client choice for well-matched and meaningful job; (f) integration of IPS within the clinical treatment team ensuring shared decision making between providers and clients; (g) personalized benefits counseling; and (h) individualized support during follow-up for as long as needed even after a competitive job is obtained. Adaptations that can be made without compromising fidelity are related to the specifics of interacting with Veterans who are recovering from PTSD and integrating IPS in a new treatment team, that is, the PTSD Clinical Treatment teams. These adaptations are based on PTSD treatment team culture and patients’ behavioral symptoms that differ from the traditional IPS integration with mental health teams that care for Veterans with serious mental illness (i.e., psychotic disorders). For example, compared to Veterans with psychotic disorders, patients living with PTSD tend to be more active in their own job development and can find a job, but are often unable to sustain employment which subsequently requires greater collaboration and partnership between the patient, IPS specialist and PTSD treatment provider.
5. What are the benchmarks of successful implementation?   a. Explain how successful implementation will be measured.   b. Describe how the impact of dissemination or implementation will be measured.   c. Include patient-perspective benchmarks as well, including health equity, barriers, and satisfaction No metrics were drafted in the study protocol to evaluate these parameters. Future implementation projects should include these measures of success: number of Veterans referred for and engaged in IPS services per month, number and duration of competitive jobs gained per participant, income earned per participant, patient satisfaction surveys, IPS fidelity scale scores, and job satisfaction scores.
6. How is the intervention perceived and used by key interested parties?   a. Consider frontline providers, local leadership, and patients.   b. Elicit feedback from interested parties about equity and diversity concerns regarding implementation of the intervention or treatment.   Note: This will include conversations and/or qualitative interviews with frontline users (overlap with and/or modified from pre-implementation questions. During on-site visits, the IPS trainers and fidelity monitor would interview frontline providers, staff, and patients to obtain firsthand information about IPS implementation, barriers, and facilitators. There were varied levels of integration of IPS within the PTSD treatment team. For example, providers, who were understanding of and embraced the recovery model and principles of IPS, would make early and ongoing referrals. However, there were providers who were initially reluctant to refer Veterans for fear that the Veteran was “not ready" to return to work or the work would exacerbate the symptoms of PTSD. These fears diminished as case examples proved the resiliency of the Veterans and strengths of individual supports provided by the IPS specialists. Some Veterans had misperceptions that returning to work would jeopardize their VA disability benefits, but these Veterans were provided education on the legal statutes that protect benefits while participating in a VA vocational rehabilitation program. The IPS trainer/fidelity monitor would provide timely information and feedback to the local site investigators and IPS specialists, which helped improve and accelerate implementation of the IPS intervention. The discussion with the providers was done in the context of the various study team meetings. The CSP Study Chair (Davis) made individual site visits to launch the study instead of holding an all-hands national start-up meeting, which helped tailor IPS implementation at the site level since she could meet with many more providers and leadership at the site level instead of only the research team at a national meeting.
7. What is the plan for assessment of the uptake and fidelity in delivery of the treatment or intervention?   a. Describe how the uptake or use of the intervention or treatment by the patients is measured (ideally, using EHR data)   b. Describe who is responsible for assessing fidelity of the treatment or intervention delivery by the provider. How will fidelity be assessed among existing providers (i.e., those not funded by the study) The only fidelity monitoring of the intervention was done during the study. No others were carried out once the study ended. The IPS trainer/fidelity monitor has over 30 years of IPS implementation experience across community mental health and VA settings. Measurement of fidelity of IPS implementation is essential and scores correlate with patient-level employment outcomes, that is, higher fidelity scores yield better employment outcomes.
8. What preliminary insight can key interested parties (patient/Veteran as well as clinical interested parties, including local-, regional- and national-level leadership) provide about barriers to sustainment (to inform phase 3)? Note: This data is best collected through qualitative interviews with key interested parties and/or a fidelity or satisfaction survey. In assessing model sustainability, feedback and insight would be obtained from involved Veteran participants and their healthcare providers. In addition, upstream of these interested parties, information from local/regional and national employment services and/or service chiefs would allow for a synthesis of barriers of not only deployment but long-term viability of the IPS model.
Phase 3. Planning for Sustainment for Effective Trials: If the intervention is found to be effective, how will the intervention be spread to new sites and sustained by interested parties? If the intervention is found to be effective, it is important for there to be sustainment planning activities in place so that the intervention is maintained and continuously provides benefits to the healthcare system. Sustainment planning will help evaluators/researchers understand how the intervention will be used in routine care once the study has ended and if there are important equity and diversity considerations needed for sustainment. Results from Phase 1 and 2 will inform clinicians and healthcare sites in understanding how to deliver the intervention protocol more effectively, make appropriate adaptations, and sustain the intervention over time. This can be written as a “toolkit” or “implementation playbook” that outlines next steps for national policy and organization changed and recommendations for provider training and delivery. This “toolkit,” built during the course of the trial, can then be deployable to other (new) sites if the intervention is found effective.
1. How can results from Phases 1 and 2 be used to develop an implementation strategy, process, or “playbook” by which interventions will be adopted and sustained in routine practice by existing providers?   a. Describe the extent to which the trial was pragmatic and its implications for sustainment (i.e., if it was not pragmatic, how will might this impact sustainment and scale-up of the intervention or treatment?   b. Describe the training required among existing and new providers to deliver the treatment or intervention   c. Describe the clinical processes required to maintain the treatment or intervention in routine practice and what additional administrative changes might be needed (e.g., addition of treatment into VA national formulary, lab tests, additional clinic time or procedures required to deliver treatment), where the intervention will take place (e.g., primary care, specialty, etc.), and how sustainable the intervention is (i.e., consider resources and staffing once the trial ends).   d. Determine which measures should be used to monitor use of the intervention or treatment in routine practice (e.g., from existing EHR data, addition of variables to the formulary/EHR etc.   e. Identify how sustainment of the intervention can be tracked over time (e.g., dashboards, surveys).   f. Consider how qualitative data from leadership or interested parties can be used to understand the factors that may help or hinder future “real-world” spread, implementation, or sustainment and how there may be unique equity and diversity considerations during these different phases   g. Describe if certain implementation strategies are more likely to sustain this intervention over time and have sustained effects over time.   h. Consider how results will inform future interested parties’ acceptance of the intervention.   i. Collect information on cost, including opportunity costs, and burden from different interested party perspectives, patient engagement, satisfaction, barriers, and health equity.   Note: Qualitative interviews or surveys with intervention-interested parties during phase 1 and phase 2 could be used to understand potential sustainment of the intervention 3 The implementation of the intervention was left to the individual VA Medical Center sites and there was no component in the proposal to address this. The Study Chair continued carrying the message of the effectiveness of this intervention through clinical care interested parties, additional research into IPS cost efficacy, other funded implementation projects, and various dissemination efforts. As shown in follow-up research study, IPS is sustainable and cost-effective with good social return on investment. Sustainability can be measured through existing data collection from VA Northeast Program Evaluation (NEPEC) as is done for IPS service delivery for the seriously mentally ill population and other PTSD programs. Qualitative data from key leadership and interested parties are important factors in real-world spread. In a resource-constrained healthcare system such as VHA, the main barriers to implementation of IPS include resource allocation (FTE) and need to replace outdated vocational rehabilitation services that have no evidence-based rationale; however, this strategy is threatening to the status quo within local VA facilities and at the national-level program office. The traditional vocational rehabilitation services place Veterans in low-wage transient jobs typically in the VA medical center for “work hardening experiences” prior to transitioning them without much follow-along support into the competitive job environment. VA facilities may have a secondary gain in having a labor pool for low-wage low-skill jobs, such as Veterans with disabilities, to meet the needs of the facility confronted with short staffing in areas such as housekeeping and grounds maintenance. These issues trigger political tensions within the national-level program office that are difficult for an expert investigator assigned to a local VA to navigate withouthigh-level interested party support.
Phases 1–3. Planning for Dissemination: How will intervention and implementation information and trial results be shared with others to increase adoption of the intervention? Throughout all 3 phases of implementation planning, teams should consider the types of information that should be disseminated, to whom information should be disseminated, and how information should be tailored to address equity and diversity considerations of different individuals and healthcare settings. Dissemination is important for increasing awareness of the intervention, offering opportunities for bidirectional communication, and accelerating the buy-in/adoption/uptake of the intervention by providers, patients, and/or healthcare systems.
1. What information should be disseminated during Phase 1 “Planning, Framing, and Aligning Interested parties” (e.g., increase intervention awareness and buy-in among interested parties)?   a. Identify to whom information should be disseminated by creating a visual display or mapping of interested parties   b. Consider dissemination as an opportunity for bidirectional communication to and from interested parties to inform initial planning This was done in the context of planning with the interested parties present at the CSP planning meetings. In hindsight, more could have been done at the national level to accelerate buy-in and uptake of the intervention.
2. What information should be disseminated during Phase 2 “Implementation Process and Data Collection” (e.g., Share implementation plans and support tools with key interested parties)?   a. Identify to whom information should be disseminated by creating a visual display or mapping of interested parties   b. Consider dissemination as an opportunity for bidirectional communication to and from interested parties to inform implementation Fidelity monitoring visits would happen semi-annually at each site. Feedback was given during/after those visits. In addition, feedback and data on this were presented to all site research personnel at the Study’s Annual Meetings. On a regular basis, participating VA Medical Centers would receive feedback from the IPS fidelity monitor as it related to the implementation of the intervention during the execution phase of the project. He would also share implementation support tools and encourage planning for post-study sustainment with facility leadership during de-briefing meetings following semi-annual IPS fidelity monitoring visits. The regional IPS mentors would use this information to provide technical assistance by weekly teleconference during bi-monthly on-site field mentoring visits. Once a year, all participating VA Medical Center personnel would be informed of such information through data reports presented at in-person Study Group meetings.
3. What information should be disseminated during Phase 3 “Planning for Sustainment for Effective Trials” (e.g., maintain priority and awareness of the intervention among key interested parties)?   a. Identify to whom information should be disseminated by creating a visual display or mapping of interested parties   b. Consider dissemination as an opportunity for bidirectional communication to and from interested parties to inform sustainment During sustainment, the actual outcomes of Veterans participating in IPS services should be shared (qualitative and quantitative outcomes) with interested parties so that awareness and support for the program can be maintained and new research findings can be considered to adapt the program so that it achieves the highest impact possible. Plans should include an executive roundtable including medical center directors/leadership, Veterans Affairs Central Office, Secretary’s office, and VA Center for Innovation.
4. Throughout all three phases, what is the plan for how and when intervention information and trial results will be disseminated, tailored, and communicated to interested parties and potential non-VA adopters? The following considerations may differ for each of the three phases:   a. Identify various “passive” publication opportunities (e.g., peer-reviewed journals, other publications)   b. Identify more “active” strategies for disseminating information and results, which have been found to be more effective in reaching key interested parties (e.g., briefings to VA local, regional and national leaders-CMO national calls, VISN meetings, program office meetings, news or social media outlets, workshops, meetings)   c. Determine if there are local or national opportunities to present trial results (e.g., professional conferences, VHA Cyber-seminars)   d. Consider how intervention materials and results can be disseminated to diverse interested parties (e.g., clinical and non-clinical audiences, patient groups, Veteran and Family Advisory Councils).   e. Develop other products that could be used to disseminate trial results (e.g., websites, toolkits)   Note: Consider developing an interested parties map to identify and engage relevant interested parties in future implementation processes. B-E will likely offer more opportunities for bidirectional communication The plan for dissemination was limited to publicity of the study results, to include peer-reviewed manuscripts, presentations at international conferences, talks by the Study leadership at various VA Medical Center facilities and outside healthcare systems, VHA Cyber-seminars, VA Center of Innovation meetings, and VA regional and local mental health summits.
5. What are the plans for using the intervention in future implementation research? Note: Consider funding from a VA QUERI center, VA Office of Research and Development/Health Services Research & Development (HSR&D) research study, VA national program office policy (use the Evidence Act), VA Diffusion of Excellence, National Institutes of Health-funded research, or foundation grants. Although this was not considered in the context of the CSP research project, the study Chair and IPS fidelity monitor went on to launch a 3-year pay-for-success social impact bond implementation of IPS at four VA Medical Centers in the Northeast area, called Veteran CARE. Dr Davis and members of the CSP Coordinating Center also conducted an HSR&D-funded cost efficacy study utilizing CSP data. Dr Davis and her work in IPS has been featured in VA Diffusion of Excellence marketplace. Dr Davis was also subsequently funded by VA Rehabilitation Research & Development to test the efficacy of IPS in a primary care setting for Veterans with a broad range of non-psychotic mental health conditions.

¹ While the tool was designed for use in both efficacy and effectiveness trials, effectiveness trials will have broader and more involved implementation methodology since they focus on answering the question: do the intervention benefits hold true in real-world clinical settings? In contrast, the tool will be more limited in efficacy trials since they focus on the answering the question: does the intervention work in a highly controlled standardized setting? Therefore, throughout the tool, we have indicated sections that may not be useful during efficacy trials.

² See Frayne SM, Pomernacki A, Schnurr PP. Women’s Enhanced Recruitment Process (WERP): Experience with Enhanced Recruitment of Women Veterans to a CSP Trial. Invited national VA HSR&D CyberSeminar, presented November 15, 2018. https://www.hsrd.research.va.gov/for_researchers/cyber_seminars/archives/video_archive.cfm?SessionID=3565

³ Denotes overlap between Planning, Framing, and Aligning Interested Parties Phase and Implementation Process Data Collection phases.