. 2022 Oct 6;6(1):e131. doi: 10.1017/cts.2022.467

Table 1.

The Implementation Planning Assessment Tool¹

Implementation Planning Assessment Tool for the Veterans Health Administration (VHA) Cooperative Studies Program (CSP) and Other Clinical Trials
Guidance: As a team, review and document responses to the overarching questions. The intention is for this tool to be completed as an iterative process and the teams and individuals can and should refer to the tool at different points in time throughout the trial.
Phase 1. Planning, Framing, and Aligning Interested Parties: How do CSP trial programs identify and align interested parties? Planning, framing, and aligning interested parties helps inform the design of the intervention to be implemented (e.g., design-for implementation, user-centered design). Clinical trial programs often include national interested parties upfront on their Executive Committees, such as national program office leads (e.g., Pharmacy Benefits Management, National Pathology & Laboratory Medicine, Patient Care Services, Clinical Services program offices). However, these programs should also consider identifying and collecting input from potential end-users at the regional (e.g., Veterans Integrated Service Network (VISN), Chief Medical officers (CMOs)) and local levels (e.g., facility Chiefs of Staff and Service Line Chiefs), and Veteran or patient users throughout the trial process. Involving interested parties at multiple levels will also help enhance equity and diversity in implementation planning and garner buy-in, at the local clinic (e.g., frontline provider, facility service line) and regional managerial levels (e.g., VISN Director, CMO).
1. What is the intervention/treatment and what are its core elements that are hypothesized to achieve its desired effect on health? Intervention: Core Elements: Desired Effect on Health:
2. What clinical issue or public health problem is the intervention trying to solve? Clinical Issue: Intervention:
3. Create a list of the key interested parties involved in the intervention/treatment (please see Table 3 for a list of potential interested parties). Consider including those with high interest in the treatment or intervention, as well as key influencers in its adoption and sustainment in routine practice over time, and those who may provide insight on equity and diversity considerations. For example, clinical and non-clinical staff, local and regional managers, national program office leads, policymakers, and patients with diverse backgrounds (race/ethnicity, gender, LGBTQ+, disability) in trials. The VA Women’s Enhanced Recruitment Process (WERP) is an example of an effort to increase women’s participation in trials.²:
4. What staff and resources do you have to support preliminary implementation planning work and delivery of the intervention/treatment? a. For planning of the study including treatment or intervention/treatment delivery. b. For capturing data on the process by which the treatment is being implemented, such as provider delivery and fidelity to the treatment (e.g., checklists of treatment core elements completed and delivered to each patient). For data capture of the treatment use by existing sources for future surveillance (e.g., Electronic Health Record (EHR), diagnostic, lab, treatment codes). For capturing interested party perspectives of the treatment (patients providers, managers, leaders), such as surveys, interview guides, software.,. Consider including Full-Time Equivalent (FTE) for these tasks. c. Consider staff FTE, protected time, training, supplies (audio recorder, headset, qualitative data analysis software), and equity and diversity of facilities or staff implementing the intervention.
5. What level of facility, regional, and national leadership support is there for the intervention (describe from high to low)? a. For facility leadership explain the type of support that leadership is providing for the intervention/treatment (e.g., FTE for clinical trial, prioritizing and spreading support for the intervention or treatment trial recruitment, etc.). Note: This step includes assessing the current leadership support at involved sites (facilities) and using strategies to increase that support as necessary¹ Qualitative interviews, focus groups, or telephone conversations are methods often used to obtain feedback from leadership. National: National and Local Site Collaboration: Local Site Support:
6. Have frontline users (clinical and non-clinical staff) provided input on the design and deployment of the intervention? Note: Qualitative interviews, focus groups, or conversations are methods often used to obtain feedback from frontline users ³
7. Which determinants framework³ will be used to identify and describe contextual factors that could influence the implementation process and quality of intervention protocol delivery? a. The framework informs data collection and analysis to identify barriers and facilitators to practice uptake. Note: Examples of determinants frameworks include Theoretical Domains Framework (TDF), Consolidated Framework for Implementation Research (CFIR) [19], and the integrated-Promoting Action on Research Implementation in Health Services (i-PARIHS) [20].
8. Who are the multi-level interested parties of the intervention (e.g., those who have an interest in using the treatment or intervention if proven effective, those who may provide insight on equity and diversity considerations of individuals and/or healthcare settings, and/or those who have influence on the policies that would foster sustainment of the treatment in routine practice if proven effective)? a. Describe how interested parties were identified and how their perspectives will be assessed over the course of the trial at each participating site (e.g., qualitative or quantitative methods). b. Identify the local and regional interested parties involved in the intervention. c. Map out who needs to take different steps (role) and when in the process for the intervention to be successfully implemented. d. Identify Veteran or patient interested parties. Interested Party Input: Site Selection: Post-selection:
9. How have multi-level interested parties who have an interest in the intervention/treatment provided input? Interested parties should include Veteran or patient perspectives as well. For all interested parties, elicit feedback about equity, diversity, inclusion, barriers, facilitators, and satisfaction with the intervention/treatment. a. Consider local, regional, national interested parties. Note: Qualitative interviews, focus groups, or conversations, advisory calls, are methods often used to obtain feedback from key interested parties. ³
10. Which national program officers and operational partners could facilitate pathways for future spread or implementation?
11. Draft an implementation plan and ensure that your implementation plan includes each of the following important components: a. The rationale for the selection of appropriate implementation science theories or frameworks. b. Methods for quantitative (e.g., intervention uptake based on existing EHR data), qualitative, or mixed methods data collection and analyses. If using mixed methods, how will the methods be integrated? c. Quantitative sampling plan for control and comparison groups and data codes and fields to capture use in the EHR. CSP implementation leads should coordinate with CSP coordinating centers to determine what data is already being collected to avoid duplication. If not applicable, describe the rationale. d. Qualitative sampling plan of patient and clinical interested parties (e.g., purposeful criterion, stratified, snowball strategies). If not applicable, describe the rationale. e. Specification of the types (quantitative or qualitative) or sources of data (e.g., primary or secondary) to be used, data accessibility, aggregate and subgroup analyses, and provisions for ensuring data quality and adherence to the study protocol.
12. What are the preliminary plans for the interventions’ sustainment, once the trial ends, if the intervention/treatment is found effective (see also Phase 3)? The plan should take into consideration any administrative or policy changes needed at the national and regional levels (e.g., formularies, labs, EHR fields, national directives, or other services policies), time, tools, and training required by clinicians at the frontline to deliver the intervention and where (e.g., primary care, specialty care clinics, Community-Based Outpatient Clinics, etc.) and Veteran time required (e.g., visits, required lab tests, medications, etc.) a. The plan should also take into consideration how pragmatic the trial is (e.g., factors that could impact use of the intervention in real-world settings such as cost, intervention deliverers, intervention recipients).
Phase 2. Implementation Process Data Collection: How will the implementation process be studied, measured, and assessed? The Implementation Process Data Collection Phase involves ascertainment of factors affecting the use of the CSP intervention or treatment at the routine practice level, notably through information on provider and patient perspectives and acceptance, implementation and intervention costs and organizational factors, and where relevant fidelity to the implementation of the intervention or treatment. This phase also involves enacting an implementation assessment plan and should include equity and diversity considerations throughout.
1. Who is part of your assessment team? Describe the amount and type of time set-aside for the assessment team, implementation lead, or others (e.g., FTE, protected time, donated).
2. Have you finalized your assessment plan (#11 in planning) to address the following? a. The rationale for the selection of appropriate implementation science theories or frameworks. b. Methods for quantitative, qualitative, or mixed methods data collections and analyses. If using mixed methods, how will the methods be integrated? c. Quantitative sampling plan for control and comparison groups for describing the implementation process and intervention uptake and use. If not applicable, describe the rationale. d. Qualitative sampling plan of patient, local-, regional- and national-level leadership, and clinical interested parties, including specific techniques and measures. If not applicable, describe the rationale. e. Specification of the kinds or sources of data to be used, data accessibility, aggregate and subgroup analyses, and provisions for ensuring data quality and adherence to the study protocol.
3. Which implementation strategies did you select to help attain successful implementation of the intervention? a. For efficacy trials, define your implementation strategies and their goals. b. For effectiveness trials, is the trial paying for providers to deliver the treatment/ intervention to the patients or relying on existing providers? How do you intend to measure: 1) fidelity to the intervention or treatment, 2) the uptake or use of the treatment (e.g., patient use or “dose” and 3) what will training, and competencies look like for existing providers once the study ends and the intervention is shown to be effective (e.g., for providers to take up the effective intervention, what level of expertise and training is optimal and what will the manual contain on how they deliver the treatment?) 4) identify the costs of training, 5) consider interested party buy-in for training c. Document data about your implementation strategy enactment that could help future sites during scale-up and spread if the intervention is found effective d. To what extent have interested parties provided input into the selection of the implementation strategies? e. Describe the dose of implementation strategies [16]. Note: Selection of implementation strategies will be based on the key barriers and key facilitators identified during pre-implementation.
4. What adaptations or resources are needed for to fit local contexts? Please note for efficacy trials, adaptations may be less relevant or applicable. a. Describe the core components of the clinical intervention or treatment that are believed to be the causal mechanisms of therapeutic change and are responsible for achieving the intervention or treatment’s desired effects (those facets of the intervention that cannot be adapted or changed). b. Explain how the intervention can be adapted without compromising fidelity. c. Consider how adaptations can address or increase equitable implementation of the innovation.
5. What are the benchmarks of successful implementation? a. Explain how successful implementation will be measured. b. Describe how the impact of dissemination or implementation will be measured. c. Include patient-perspective benchmarks as well, including health equity, barriers, and satisfaction.
6. How is the intervention perceived and used by key interested parties? a. Consider frontline providers/staff, local leadership, and patients. b. Elicit feedback from interested parties about equity and diversity concerns regarding implementation of the intervention or treatment. Note: This will include conversations and/or qualitative interviews with frontline users (overlap with and/or modified from pre-implementation questions.
7. What is the plan for assessment of the uptake and fidelity in delivery of the treatment or intervention? a. Describe how the uptake or use of the intervention or treatment by the patients is measured (ideally, using EHR data) b. Describe who is responsible for assessing fidelity of the treatment or intervention delivery by the provider. How will fidelity be assessed among existing providers (i.e., those not funded by the study)
8. What preliminary insight can key interested parties (patient/Veteran as well as clinical interested parties, including local-, regional- and national-level leadership) provide about barriers to sustainment (to inform phase 3)?
Phase 3. Planning for Sustainment for Effective Trials: If the intervention is found to be effective, how will the intervention be spread to new sites and sustained by interested parties? If the intervention is found to be effective, it is important for there to be sustainment planning activities in place so that the intervention is maintained and continuously provides benefits to the healthcare system. Sustainment planning will help evaluators/researchers understand how the intervention will be used in routine care once the study has ended and if there are important equity and diversity considerations needed for sustainment. Results from Phase 1 and 2 will inform clinicians and healthcare sites in understanding how to deliver the intervention protocol more effectively, make appropriate adaptations, and sustain the intervention over time. This can be written as a “toolkit” or “implementation playbook” that outlines next steps for national policy and organization changed and recommendations for provider training and delivery. This “toolkit,” built during the course of the trial, can then be deployable to other (new) sites if the intervention is found effective.
1. How can results from Phases 1 and 2 be used to develop an implementation strategy, process, or “playbook” by which interventions will be adopted and sustained in routine practice by existing providers? a. Describe the extent to which the trial was pragmatic and its implications for sustainment (i.e., if it was not pragmatic, how will might this impact sustainment and scale-up of the intervention or treatment? b. Describe the training required among existing and new providers/staff to deliver the treatment or intervention c. Describe the clinical processes required to maintain the treatment or intervention in routine practice and what additional administrative changes might be needed (e.g., addition of treatment into VA national formulary, lab tests, additional clinic time or procedures required to deliver treatment), where the intervention will take place (e.g., primary care, specialty, etc.), and how sustainable the intervention is (i.e., consider resources and staffing once the trial ends). d. Determine which measures should be used to monitor use of the intervention or treatment in routine practice (e.g., from existing EHR data, addition of variables to the formulary/EHR, etc. e. Identify how sustainment of the intervention can be tracked over time (e.g., dashboards, surveys). f. Consider how qualitative data from leadership or interested parties can be used to understand the factors that may help or hinder future “real-world” spread, implementation, or sustainment and how there may be unique equity and diversity considerations during these different phases g. Describe if certain implementation strategies are more likely to sustain this intervention over time and have sustained effects over time. h. Consider how results will inform future interested parties’ acceptance of the intervention. i. Collect information on cost, including opportunity costs, and burden from different interested party perspectives, patient engagement, satisfaction, barriers, and health equity. Note: Qualitative interviews or surveys with intervention-interested parties during phase 1 and phase 2 could be used to understand potential sustainment of the intervention3
Phases 1–3. Planning for Dissemination: How will intervention and implementation information and trial results be shared with others to increase adoption of the intervention? Throughout all 3 phases of implementation planning, teams should consider the types of information that should be disseminated, to whom information should be disseminated, and how information should be tailored to address equity and diversity considerations of different individuals and healthcare settings. Dissemination is important for increasing awareness of the intervention, offering opportunities for bidirectional communication, and accelerating the buy-in/adoption/uptake of the intervention by providers, patients, and/or healthcare systems.
1. What information should be disseminated during Phase 1 “Planning, Framing, and Aligning Interested parties” (e.g., increase intervention awareness and buy-in among interested parties)? a. Identify to whom information should be disseminated by creating a visual display or mapping of interested parties b. Consider dissemination as an opportunity for bidirectional communication to and from interested parties to inform initial planning
2. What information should be disseminated during Phase 2 “Implementation Process and Data Collection” (e.g., Share implementation plans and support tools with key interested parties)? a. Identify to whom information should be disseminated by creating a visual display or mapping of interested parties b. Consider dissemination as an opportunity for bidirectional communication to and from interested parties to inform implementation
3. What information should be disseminated during Phase 3 “Planning for Sustainment for Effective Trials” (e.g., maintain priority and awareness of the intervention among key interested parties)? a. Identify to whom information should be disseminated by creating a visual display or mapping of interested parties b. Consider dissemination as an opportunity for bidirectional communication to and from interested parties to inform sustainment
4. Throughout all three phases, what is the plan for how and when intervention information and trial results will be disseminated, tailored, and communicated to interested parties and potential non-VA adopters? The following considerations may differ for each of the three phases: a. Identify various “passive” publication opportunities (e.g., peer-reviewed journals, other publications) b. Identify more “active” strategies for disseminating information and results, which have been found to be more effective in reaching key interested parties (e.g., briefings to VA local, regional and national leaders-CMO national calls, VISN meetings, program office meetings, news or social media outlets, workshops, meetings) c. Determine if there are local or national opportunities to present trial results (e.g., professional conferences, VHA Cyber-seminars) d. Consider how intervention materials and results can be disseminated to diverse interested parties (e.g., clinical and non-clinical audiences, patient groups, Veteran and Family Advisory Councils). e. Develop other products that could be used to disseminate trial results (e.g., websites, toolkits, playbooks) Note: Consider developing an interested parties map to identify and engage relevant interested parties in future implementation processes. B-E will likely offer more opportunities for bidirectional communication
5. What are the plans for using the intervention in future implementation research? Note: Consider funding from a VA QUERI center, VA Office of Research and Development/Health Services Research & Development (HSR&D) research study, VA national program office policy (use the Evidence Act), VA Diffusion of Excellence, National Institutes of Health-funded research, or foundation grants.

While the tool was designed for use in both efficacy and effectiveness trials, effectiveness trials will have broader and more involved implementation methodology since they focus on answering the question: do the intervention benefits hold true in real-world clinical settings? In contrast, the tool will be more limited in efficacy trials since they focus on the answering the question: does the intervention work in a highly controlled standardized setting? Therefore, throughout the tool, we have indicated sections that may not be useful during efficacy trials.

See Frayne SM, Pomernacki A, Schnurr PP. Women’s Enhanced Recruitment Process (WERP): Experience with Enhanced Recruitment of Women Veterans to a CSP Trial. Invited national VA HSR&D CyberSeminar, presented November 15, 2018. https://www.hsrd.research.va.gov/for_researchers/cyber_seminars/archives/video_archive.cfm?SessionID=3565 https://www.hsrd.research.va.gov/for_researchers/cyber_seminars/archives/video_archive.cfm?Sessio

Denotes overlap between Planning, Framing, and Aligning Interested Parties Phase and Implementation Process Data Collection phases.