Abstract
Background
Measuring the health, economic, and cultural gains generated by scientific investments is crucial to reducing waste and improving quality of care. To date, there is no comprehensive framework for assessing the multi-faceted contributions of implementation and quality improvement sciences towards quality, cost, and patient and provider experiences in health systems.
Objective
We describe the Quality Enhancement Research Initiative (QUERI) Impact Framework and its application to QUERI investments.
Methods
The QUERI Impact Framework adapts and expands on metrics from the National Academy of Medicine, incorporating lessons learned from QUERI initiatives. The cross-cutting impact metrics reflect QUERI’s strategic methodology across five domains of impact (Alignment, Commitment, Tailoring, Informing the field, Observing healthcare changes and generating New questions/projects or ACTION). Key impact metrics, including the number of implementation facilities, number of staff trained, and number of patients served, were derived directly from health system performance plan goals. QUERI applied the Framework by conducting iterative rapid assessments of impacts for QUERI Program centers, which are implementation laboratories that support 3–7 initiatives aligned with a cross-disciplinary goal addressing a national priority.
Key Results
From October 2015 to September 2019, QUERI Programs supported implementation of 49 evidence-based practices and promising innovations across 465 facilities, including 15 facilities that are experiencing quality gaps. As part of these implementation efforts, the programs worked with 71 operations partners to develop 71 tools/toolkits/manuals and support training of 5147 VA staff, serving 250,159 Veterans.
Conclusions
The QUERI Impact Framework aligns multiple stakeholders at different levels of a health system around common metrics, which cross implementation science and quality improvement boundaries. The Framework supports a comprehensive assessment of the short-term and distal impacts of implementation efforts in a health system, allowing both research and operations leadership to understand the value of implementation and quality improvement investments to inform program and policy decisions.
KEY WORDS: Research impact, implementation science, quality improvement, knowledge translation, Veteran health
BACKGROUND
Science impact assessment is crucial for understanding the benefits and value of research investments and dissemination activities toward improving public health and reducing research waste.1 A key question is whether research discoveries result in evidence-based practices (EBPs) and implementation or quality improvement efforts that lead to improvements in population health. The Foundations for Evidence-based Policymaking Act of 2018—which requires federal Cabinet-level agencies, including the US Department of Health and Human Services and the Department of Veterans Affairs (VA), to use scientifically generated evidence to justify budget investments—has accelerated the need to determine whether research evidence is making a substantial impact on policies and programs.2 However, impact assessment remains a challenge and is complicated by the need to identify and systematically track measures that accurately capture impact, link upstream research with changes in distal health outcomes, and attribute specific organizational efforts given the overall body of evidence that led to the development of the innovation.3
Located in the VA Office of Research & Development (ORD), the mission of the Quality Enhancement Research Initiative (QUERI) is to improve the quality and delivery of care for US Military Veterans through translating research findings into practice. QUERI’s strategic methodology is threefold: (1) implement evidence-based practices (EBPs) and promising innovations into routine care; (2) evaluate programs and policies to optimize care; (3) promote the use of implementation and quality improvement science methods across the VA. Since its inception in 1998, QUERI has funded more than 500 peer-reviewed systematic investigations focused on transforming Veteran care using scientifically supported methods and has grown to support a national network of 200+ investigators. To maximize scientific impact, QUERI promotes research-operations partnerships between investigators distributed at VA medical centers across the US and national, regional, and local VA operations leaders. VA operations partners are senior financial, strategy, and/or clinical leaders who work in a National Program Office (e.g., Executive Director, Office of Mental Health and Suicide Prevention), a regional health system (e.g., Chief Medical Officer of a Veterans Integrated Service Network), or a medical center (e.g., Medical Center Director) and can provide guidance and support to ensure QUERI project goals are aligned with health system needs and improve uptake and feasibility of the EBP or promising innovation (hereafter referred to collectively as EBP). These multi-level partnerships are essential for obtaining buy-in and tailoring EBPs to meet regional and local needs while ensuring alignment with national priorities and standards.
This paper describes the QUERI Impact Framework and its implementation in the VA. Developed in response to an increased demand among VA leadership to understand the real-world impacts of VA-funded programs, the QUERI Impact Framework strives to provide a pragmatic approach for measuring the overall health, economic, policy, and cultural impacts of implementation and quality improvement sciences.
METHODS
Methods and data ascertainment for analyses were considered non-research and did not require institutional review board approval per ORD Program Guide 1200.21.
Ascertainment of Evidence Supporting the QUERI Impact Framework
Established in 1998 as an intramural funding program to translate research evidence into clinical practice, QUERI initially focused on disseminating EBPs that had evidence demonstrating improved outcomes for diseases prevalent among Veterans (e.g., heart disease, spinal cord injury), and assessments of project progress and impacts typically included project completion and publications.4 In response to changes in VA priorities and an external evaluation of QUERI in 2014, QUERI began partnering with national VA leadership to address cross-cutting clinical priorities (e.g., care coordination, patient safety) through the application of implementation strategies, which support providers in using an innovation.5, 6 The 2016–2020 QUERI Strategic Plan resulted in the establishment of 15 QUERI Programs to support implementation efforts addressing clinical variation across the VA.7 As a result, QUERI Impact Framework metrics were created through an iterative process of development, deployment, and refinement (Table 1). With the shift from disease-focused investigator-initiated projects to interdisciplinary problem-focused centers where goals are driven by the health system, QUERI began to benchmark progress using key measures derived from the National Academy of Medicine Degrees of Impact Framework.7, 8 The QUERI Impact Framework expands on QUERI Program impact domains and incorporates lessons learned from QUERI initiatives, VA transformational efforts since 2015, and 2021–2025 QUERI strategic planning.7
Table 1.
Multiple Levels of Influence Leading to the Development of the QUERI Impact Framework
| 2014–2015 | 2017–2018 | 2018 | 2018–2019 | |
|---|---|---|---|---|
| Level of influence | Passage of Choice Act and resulting national reorganization | Release of VA Medical Center Director and VISN Director Performance Plan shifting more decision-making toward regional and local levels | New Office of Research & Development (ORD) priorities | 2021–2025 QUERI strategic planning |
| New QUERI partnerships | National operations partnerships with clinical- or policy-based VA National Program Offices | Regional and local operations partnerships with VISN and facility leadership | National research service partnerships | Multi-level bi-directional partnerships with various stakeholders across VA |
| Effect on QUERI Impact Framework | New 2015 QUERI Program Impact Metrics derived from National Academy of Medicine Degrees of Impact Framework | Additional metrics derived from VA performance plan goals | Alignment with ORD priorities, cost, and provider engagement metrics | New non-monetary commitment, geographic reach, sustainability, quality of care, and culture metrics |
The 2021–2025 QUERI strategic planning process—focused on updating the 2016–2020 QUERI Strategic Plan—provided more depth to the QUERI Impact Framework by incorporating perspectives from QUERI investigators and VA leadership. In November 2018, VA operations leaders were sent an online survey to capture their priorities, involvement with QUERI-funded initiatives, and perceptions of QUERI, including QUERI’s responsiveness to VA priorities and its impact on Veteran care. Through purposeful sampling to ensure diverse geographic representation, 2–3 leaders from each regional health system, Veterans Integrated Service Network (VISN), were invited to participate in semi-structured phone interviews. In March 2019, an online survey was sent out to QUERI-funded teams, and similarly a sample of QUERI investigators were identified to ensure geographic and functional diversity and invited to participate in semi-structured telephone interviews. QUERI investigators were asked about successes and challenges related to their QUERI-funded activities, satisfaction with QUERI, and QUERI’s overall strengths and opportunities for improvement. The results of strategic planning refined the QUERI Impact Framework, promoting leaner domains and metrics and underscoring the importance of understanding the impact of non-research activities (e.g., developing and maintaining multi-level partnerships, ensuring sustainability of an EBP), measuring the impact of partnerships beyond monetary investments, assessing the geographic reach and scope of QUERI’s implementation efforts across VISNs, benchmarking impact using quality of care metrics, and understanding the perceptions of QUERI across stakeholders at the local, regional, and national levels of the health system.
QUERI Impact Framework
The QUERI Impact Framework emphasizes alignment with research and operations priorities and formation of bi-directional multi-level partnerships, which are crucial for changing clinical practice and healthcare policy and, ultimately, improving patient care and health outcomes. Table 2 describes potential indicators across five degrees of impact (Alignment, Commitment, Tailoring, Informing the field, Observing healthcare changes and generating New questions/projects or ACTION).
Table 2.
QUERI Impact Framework
| Domain and degrees of impact | Potential indicators |
|---|---|
| Alignment with multi-level priorities: How is the initiative directly linked to health system operations and research priorities? |
1a. Research priorities Research priority being addressed*; Number of research innovations implemented in routine care; Number of in-house research innovations implemented in routine care 1b. Health system operations priorities Health system priority being addressed*; Identified metrics derived from priorities and health system goals*; Total number of distinct partners; Scope and depth of partnerships* |
| Commitment: How effective is the research-operations partnership? |
2. Administrative research and operations resources Amount of funding provided by partners; Non-monetary resources contributed by partners*; Description of how these resources support shared goals* 3. Investigator-developed tools and strategies Type of implementation strategy used*; Innovations to implementation strategy for facilities that have quality gaps in clinical area being addressed*; Evaluation design to examine strategy or compare the effectiveness of two implementation strategies (e.g., randomized stepped wedge)*; Approach to continuous learning and improvement throughout implementation process*; Number and type* of products developed; Number of patents awarded |
| Tailoring to local context: To what extent were specific strategies and tools adopted into routine practice? |
4. Direct proximate impacts Number and type* of implementation facilities; Number and type* of implementation facilities that have quality gaps in clinical area being addressed; Number of providers trained and/or using EBP; Number of patients served by EBP |
| Informing the field: How are the project results being communicated to key stakeholders and organizations? |
5a. Extent to which results were disseminated to health system leadership and other operations stakeholders Description of briefings/presentations to health system leadership and other key stakeholders with decision-making or financial authority* 5b. Extent to which results were disseminated to research stakeholders Number of published journal articles; Number of conference presentations; List of citations 5c. Extent to which results were disseminated externally to the public Number of articles published on national media outlets; Number of mentions in social media 6. Extent to which there was additional scale up and spread beyond original implementation facilities Additional resources to scale up and spread EBP*; Number of additional sites implementing EBP/tool/strategy (beyond the original sites that were directly part of the initial implementation effort) |
| Observing healthcare changes and generating New questions and projects: What improvements in quality of care and health outcomes, policy, and/or culture were observed? Were new analyses/projects launched as a result of the effort? |
7. Sustainability Cost-benefit calculations; Cost of implementing intervention; Cost of implementation strategy; Return on investment (ROI) 8. Quality of care and health outcomesHealth outcomes of patients*; Quality of care metrics* 9. Policy Number of citations in clinical guidelines; Number of citations in policy documents 10. Culture Observed changes in provider engagement*; Number and range* of new quality enhancement projects or evaluation analyses launched or expanded |
*Qualitative metric
Alignment with Multi-Level Priorities
Alignment with research and operations partners and national priorities ensures practical strategies and tools are developed to achieve health system performance objectives while maximizing scientific impact. This participatory emphasis is captured in the Alignment Domain focused on priorities identified by stakeholders at the local, regional, and national levels. Examples of national VA priorities include reduction of unwarranted variation across integrated clinical and operational service lines and improving access to services that support and encourage lifelong health and wellness. In the context of these priorities, VISN leadership has nominated actionable priorities, such as improving access to medication-assisted treatment for opioid use disorder, for QUERI implementation support.9
Commitment
The Commitment Domain examines the productivity of collaborations in contributing to the success of the implementation effort, emphasizing joint investment and risk-sharing. In providing tangible resources, researchers and operations leaders form bi-directional strategic partnerships grounded in a shared vision and goals, a collaborative governance structure, and flexible incentives. Operations partners show commitment through contributing funding, personnel, data, and other resources to support the implementation effort, and researchers bring their expertise in developing scientifically supported tools and strategies. Examples of resources that operations leaders have provided for QUERI initiatives include funding for equipment, travel support for site visits, and protected time for non-research personnel (e.g., pharmacists). Dashboards, clinical decision support tools, training manuals, and toolkits are some examples of products developed by QUERI investigators to support the scale up and spread of EBPs.
Tailoring to Local Context
The Tailoring Domain focuses on proximate impacts linked to health system performance plan goals, promoting a shared understanding across stakeholders for benchmarking success toward achieving goals. QUERI tracks metrics derived from the VA Medical Center Director and VISN Director Performance Plan, including the number of facilities where the EBP has been implemented, the number of VA staff trained in the EBP, and the number of Veterans served by the EBP.
Like other US health systems that span multiple geographic regions, VA faces challenges with addressing clinical variation across facilities and health disparities among specific Veteran groups.10 Because QUERI investigators are experts in using implementation strategies to enhance the uptake of EBPs for facilities with complex needs, QUERI implementation teams are encouraged to provide tailored support for specific subpopulations (e.g., Veterans residing in rural areas) and facilities that are experiencing quality gaps. These more intensive efforts, including the innovative strategies used in working with facilities experiencing quality gaps, are tracked separately.
Informing the Field
Like the National Academy of Medicine (formerly Institute of Medicine) Degrees of Impact Framework and AAMC/RAND metrics, the QUERI Impact Framework measures dissemination to stakeholders with direct decision-making and financial authority and publications and presentations at academic conferences.8, 11 Capturing the number of published journal articles is crucial because it ensures investigators, who are located at institutions that often tie academic promotion to the number of high impact journal publications and conference presentations produced from research, are encouraged to lead QUERI-funded initiatives. Additionally, the QUERI Impact Framework assesses the effectiveness of these dissemination efforts by measuring the scale up and spread of the EBP beyond the original implementation facilities.
Observing Healthcare Changes and Generating New Projects
The fifth domain, focused on long-term impacts, involves monitoring the sustainability of the EBP and long-term changes in quality of care, policy, and culture across the health system. Based on the results of the 2021–2025 strategic planning where VISN and facility leadership emphasized the importance of measuring impact using quality of care metrics, QUERI Programs were encouraged to identify relevant VA performance metrics, provider experience data from the VA All Employee Survey, and other publicly reported quality metrics identified by the National Quality Forum and the Centers for Medicare and Medicaid Services (CMS).12, 13 Understanding whether an expansion or development of new projects emanated from the original initiative can help signal a cultural shift in health system leadership’s and other key stakeholders’ perceptions of the value of implementation and quality improvement methods in improving care. Furthermore, the 2021–2025 QUERI strategic planning highlighted the importance of mentoring the next generation of QUERI investigators and capturing their impact by measuring the volume and quality of products developed by early career investigators to improve patient care through the application of implementation and quality improvement methods. These long-term impacts will be systematically captured starting with the 2021–2025 QUERI Programs, which are undergoing peer review in 2020.
Application of the QUERI Impact Framework
Iterative Process for Collecting Impacts
In 2018, QUERI began deploying an iterative process of collecting impacts across the continuum of funding, including at the point of application. The inclusion of anticipated impacts as an application requirement encouraged QUERI investigators and key stakeholders to consider the potential impact of their work on improving care for Veterans and, with a Veteran-centered perspective, together identify resources and metrics to optimize impact. The biannual collection of impacts throughout the funding award period allows for ongoing evaluation and reflection so adjustments can be made to share learnings and enhance impact.
Applying the Framework to the QUERI Programs
Table 3 describes QUERI’s cross-cutting metrics, which reflect QUERI’s strategic methodology focused on implementation, evaluation, and dissemination. Funded on 5-year cycles and led by VA investigators, QUERI Program centers support 3–7 implementation- and/or evaluation-focused initiatives that reflect a cross-disciplinary impact goal aligned with VA priorities. Since the QUERI Programs have similar structures, start dates, and implementation strategy goals, the following impact analysis reviews the in-progress impacts of QUERI Program initiatives that are utilizing at least one discrete implementation strategy to support implementation of an EBP.16 Data were abstracted from Fiscal Year 2019 reports, which include cumulative impacts from the start of the QUERI Program through September 30, 2019.
Table 3.
QUERI Program Center Impacts
| Domain and degrees of impact | QUERI metrics | QUERI program center reported impacts |
|---|---|---|
| Alignment with Multi-Level Priorities: How is the initiative directly linked to health system operations and research priorities? |
1a. Research priorities Office of Research & Development (ORD) priority being addressed Number of evidence-based practices (EBPs) implemented Number of EBPs derived from ORD research implemented 1b. Health system operations priorities National VA priority being addressed Metrics derived from VA performance plan goalsNumber of distinct partners across geographical areas and service lines Role and involvement level of specific individuals in the initiative |
1a. Research priorities All QUERI Program initiatives aligned with ORD priority to move more research into real-world practice 49 EBPs 37 (76%) EBPs derived in part from ORD research 1b. Health system operations priorities All QUERI Program initiatives linked to one or more national priorities Patient engagement (e.g., awareness of diagnosed conditions) Utilization (e.g., appointment completion, screening and diagnosis of specific conditions) Caretaker and patient experiences (e.g., access, decision-making process, pain score) Provider engagement (e.g., overall satisfaction with process) Quality (e.g., error rates, rate of adverse events, hospitalization rates) Health outcomes (e.g., specific disease outcomes, weight loss, reduction in inappropriate prescriptions)Cost (e.g., cost of intervention)62 national partners representing 37 unique organizational and/or patient care service areas9 regional partners representing 7 VISNs |
| Commitment: How effective is the research-operations partnership? |
2. Administrative research and operations resources Operations partner funding commitment Tangible operations resources provided 3. Investigator-developed tools and strategies Type of strategy used Innovations and/or adaptations to implementation strategy, particularly for facilities experiencing quality gaps Evaluation design for examining implementation strategy effectiveness Description of learning collaborative or other methods used to share learnings and enhance implementation efforts Number and type of products developed by QUERI investigators |
2. Administrative research and operations resources $5,420,074 Nearly all national operations partners provided non-monetary support 3. Investigator-developed tools and strategies Facilitation* (n = 22; 45%) most commonly used, followed by audit and feedback† (n = 14; 29%) Adaptations included more intensive support and feedback, increased facilitation, recruitment of a local champion, integration of clinical reminders 31% (n = 15) of initiatives have an evaluation plan to examine effectiveness of implementation strategy versus usual care/standard implementation or comparing two implementation strategies 11 of the 15 QUERI Programs have a Learning Collaborative or other methods used to share learnings and enhance implementation efforts across projects 71 tools, toolkits, and manuals |
| Tailoring to local context: To what extent were specific strategies and tools adopted into routine practice? |
4. Direct proximate impacts Number and type of facilities implementing EBP/tool/strategy Number and type of facilities with quality gaps implementing EBP/tool/strategy Number of VA staff trained in EBP/tool/strategy or number of VA staff no longer using low-value approach (for de-implementation) Number of Veterans served by EBP/tool/strategy Special populations served by EBP/tool/strategy |
4. Direct proximate impacts 465 facilities across 18 VISNs 15 facilities with quality gaps 5147 VA staff trained 250,159 Veterans served 1 QUERI Program focuses on rural Veterans 1 QUERI Program tailors EBPs to women Veterans 2 QUERI Programs focus on vulnerable populations (e.g., homeless Veterans, Veterans at risk of functional decline) |
| Informing the field: How are the project results being communicated to key stakeholders and organizations? |
5a. Extent to which results were disseminated to health system leadership and other operations stakeholders Description of briefings held in which goals or results were directly presented to operations partners, Veterans Service Organizations, Veterans advisory councils, or other organizations with decision-making or financial authority 5b. Extent to which results were disseminated to research stakeholdersNumber of published journal articles directly stemming from QUERI-funded initiativesDescription of conference presentations directly stemming from QUERI-funded initiatives List of citations directly stemming from QUERI-funded initiatives5c. Extent to which results were disseminated externally to the public Number of articles published on national media outlets6. Extent to which there was additional scale up and spread beyond original participating facilitiesWhether EBP was submitted to Diffusion MarketplaceNumber of additional sites implementing EBP/tool/strategy (beyond the original sites that were directly part of the initial implementation effort) |
5a–c. Extent to which results were disseminated to health system leadership and other operations stakeholders, research stakeholders, and externally to the public All initiatives presented multiple times to health system leadership271 published journal articles 320 presentations at non-VA conferences/meetings 6. Extent to which there was additional scale up and spread beyond original participating facilities 15 submissions to Diffusion of Excellence (DEI) Shark Tank or Diffusion Marketplace 2 EBPs selected as semifinalists in DEI Shark Tank 1 EBP designated as gold status practice for national diffusion across VA |
| Observing healthcare changes and generating New questions/projects: What improvements in quality of care and health outcomes, policy, and/or culture were observed? Were new analyses/projects launched as a result of the effort? |
7. Sustainability Development of a business case analysis, including a data-driven evaluation plan estimating costs of implementing EBPs and ongoing maintenance of EBP 8. Quality of care and health outcomes Observed changes based on relevant quality measures or data from the patients touched by implementation effort 9. Policy Degree in which findings directly lead to the creation or improvement of policy or clinical practice guidelines and/or were used to inform US government testimony or panel meetings of national organizations 10. Culture Observed changes on employee and specifically provider engagement Number of products (e.g., toolkit, demonstration project) developed by mentees Number and depth of new quality enhancement projects or evaluation analyses launched or expanded in direct response to a specific, policy, or clinical question from VA local, regional, or national leadership |
7–10. Sustainability, quality of care and health outcomes, policy, culture 9 QUERI Programs conducting an economic analysis Most QUERI Programs reported it was too early for observed long-term changes in relevant quality and provider engagement measures and policy 30 initiatives report new quality enhancement projects or evaluation analyses |
*Often utilized for facilities with complex needs, facilitation is a multi-faceted interpersonal approach for overcoming barriers to implement an intervention14
†Audit and feedback involves changing provider behavior and improving clinical guideline adherence through delivery and feedback of quality performance data15
RESULTS
The 15 QUERI Programs have or are currently supporting implementation of 49 EBPs. Table 3 summarizes aggregate QUERI Program impacts.
Alignment with Multi-Level Priorities
All QUERI Programs align with ORD’s priority of translating more research into real-world settings with 76% of initiatives (n = 37) reporting that the evidence behind the intervention being delivered and/or implementation strategy being deployed stemmed from VA-funded peer-reviewed awards, published journal articles, and/or systematic reviews. QUERI Programs were aligned with at least one VA priority, reporting a total of 62 national partnerships representing 37 unique organizational and/or patient care service areas (e.g., mental health, pharmacy benefits) and 9 regional partnerships (e.g., VISN Chief Medical Officer) with 7 different VISNs. Seven QUERI Programs leveraged multi-level partnerships with at least one national and one regional partner. The nature of operations partner involvement ranged from consultation (e.g., providing input on implementation efforts) to deeper collaborations (e.g., co-authoring manuscripts).
Commitment
Nine (18%) initiatives reported a total of $5,420,074 in monetary support from operations partners. Nearly all QUERI initiatives reported in-kind operations support, such as dedicated staff time, data acquisition and programming, access to provider networks, and facility recruitment support.
Tailoring to Local Context
QUERI investigators supported implementation efforts across a variety of settings (e.g., outpatient clinic, community living center) touching all VISNs and 15 sites experiencing quality gaps. QUERI Programs trained 5147 health professionals, including physicians, nurses, pharmacists, psychologists, care managers, peer specialists, technicians, coordinators, and directors.
Informing the Field
Knowledge generated through the implementation effort was disseminated to operations leaders through regularly scheduled meetings, research stakeholders through 271 journal articles, and external organizations and other members of the public through 320 presentations at non-VA conferences/meetings (e.g., Academy Health research meetings).
Moreover, 15 (31%) QUERI initiatives submitted their EBP to the VA’s Diffusion of Excellence Shark Tank or Diffusion Marketplace, which are VA programs that support the scale up and spread of innovations submitted by VA employees and selected by subject matter experts and VA leadership.
Observing Healthcare Changes and Generating New Projects
Among the 9 QUERI Programs conducting an economic analysis, most are determining the cost of the EBP and/or estimating the cost of the specific implementation strategy used. The 30 projects/analyses launched in response to operations questions ranged in size and effort, including brief consultation, workgroup participation, systematic review, analysis and policy recommendations, program evaluation, metric development, and toolkit creation and dissemination. Although it was early for most QUERI Programs to demonstrate changes in Veteran health outcomes, some demonstrated clinical improvements, such as increases in disease screening and appropriate diagnosis referrals, positive changes in clinician workflow, increased provider satisfaction, improved Veteran satisfaction, and cost savings from avoided hospitalizations. QUERI Programs shaped policy and/or operations activities in a variety of ways, including refining a tool or program to optimize national rollout and informing decision-making for national or regional adoption of a program or mandate. For example, two EBPs—an inpatient walking program to improve mobility and a data-driven approach to enhance the quality of care for Veterans with transient ischemic attack (TIA) and minor stroke—were selected by VISN Directors for scale up and spread across the region.
DISCUSSION
To our knowledge, there is no comprehensive impact framework for measuring the multi-faceted contributions of implementation and quality improvement sciences in real-world settings. Impact assessment is crucial to ensure positive benefit across different levels of a health system. With its participatory emphasis and focus on capturing health, policy, economic, and cultural impacts using a combination of quantitative and qualitative metrics, the QUERI Impact Framework strives to provide a pragmatic approach for measuring the multi-level contributions of implementation and quality improvement sciences. In capturing all stages of implementation—pre-implementation, implementation, and sustainment—the QUERI Impact Framework seeks to enhance the long-term impact of research by ensuring sustainment of the EBP after initial implementation support and funding end.9
The QUERI Impact Framework builds upon previous impact frameworks, such as the Centers for Disease Control and Prevention’s Science Impact Framework, and assesses impact on quality, cost, and patient and provider experiences across all implementation activities, not solely research efforts.17 A unique feature of the QUERI Impact Framework is its central focus on alignment of stakeholders around multi-level priorities and metrics as health systems continue moving toward data-driven consumer-centered care. The ACTION Framework includes research dissemination impacts but places a greater emphasis on multi-level partnerships and broader societal impacts, which align with QUERI’s mission to improve Veteran health by utilizing implementation strategies to accelerate improvements in quality of care. Developing and maintaining multi-level partnerships are crucial non-research activities that require significant effort and are essential for successful implementation. Maintaining productive research-operations partnerships, including the importance of forming a coalition to insulate against organizational changes (e.g., leadership and provider turnover), embracing flexibility to meet operations partner needs and timelines (e.g., sharing interim data), and making connections to operations partner priorities and health system performance measures, is essential to ensure buy-in and maintain strong partnerships. Information collected on impacts is used to engage health system leadership, respond to Congressional requests for information, inform funding requests for applications, develop resources that support QUERI investigators, and evaluate opportunities for collaborations and outreach. Describing recent impacts has been important for continued involvement of health system leadership, whose timeline is often shorter than a typical research project’s. The continuous focus on impact from the point of application through the funding period also allows for rapid learning and translation.
Like other funders, QUERI grapples with identifying common metrics for implementation and quality improvement initiatives, which encompass a range of clinical areas, and in linking distal outcomes to implementation efforts given the dynamic interconnected nature of a health system and the extensive body of evidence that leads to the development of an EBP and implementation strategy. In response to QUERI investigator feedback gathered during the strategic planning process, the range of QUERI impact metrics was narrowed to 10 categories and focused on measuring the contribution of QUERI initiatives, not attribution.
Systematically capturing the impacts of scientific investments is crucial for ensuring funds are directed toward activities that yield the maximum impact based on organizational priorities. The QUERI Impact Framework proposes an approach to understand the impact of implementation and quality improvement sciences in improving quality and optimizing value in a resource-constrained health system. The Framework can be applied in other health systems and research organizations to align multiple diverse stakeholders around cross-cutting goals and metrics.
Funding Information
This work was supported by the US Department of Veterans Affairs, Veterans Health Administration, Health Services Research & Development Service.
Compliance with Ethical Standards
Conflict of Interest
The authors declare that they do not have a conflict of interest.
Disclaimer
The views expressed are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs or the US Government.
Footnotes
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Hughes A, Martin B. Enhancing impact: the value of public sector R&D. CIHE & UK Innovation Research Center, available at: http://www.ncub.co.uk/index.php?option=com_docman&view=download&category_slug=publications&alias=38-enhancing-impact-the-value-of-public-sector-r-d&Itemid=2728. Accessed August 23, 2019.
- 2.Foundations for Evidence-Based Policymaking Act of 2018. 2018. Available at: https://www.congress.gov/bill/115th-congress/house-bill/4174. Accessed August 23, 2019.
- 3.Greenhalgh T, Raftery J, Hanney S, Glover M. Research impact: a narrative review. BMC Med. 2016;14:78. doi: 10.1186/s12916-016-0620-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Feussner JR, Kizer KW, Demakis JG. The Quality Enhancement Research Initiative (QUERI): from evidence to action. Med Care. 2000;38(6 Suppl 1):I1–6. doi: 10.1097/00005650-200006001-00001. [DOI] [PubMed] [Google Scholar]
- 5.Veterans Access, Choice and Accountability Act of 2014 (“Choice Act” US Public Law 113–146). 2014. Available at: https://www.congress.gov/113/plaws/publ146/PLAW-113publ146.pdf. Accessed August 12, 2019.
- 6.Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8:139. doi: 10.1186/1748-5908-8-139. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Kilbourne AM, Elwy AR, Sales AE, Atkins D. Accelerating research impact in a learning health care system. Med Care. 2017;55(Suppl 7 Suppl 1):S4–S12. doi: 10.1097/MLR.0000000000000683. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Dzau, V. An IOM in Transition Seizing the Opportunity to Shape the Future. 2014. Institute of Medicine, available at: http://www.nationalacademies.org/hmd/~/media/Files/About%20the%20IOM/2014/PresidentsAddress2014.pdf?la=en. Accessed September 2, 2019.
- 9.Kilbourne AM, Goodrich DE, Miake-Lye I, Braganza MZ, Bowersox NW. Quality Enhancement Research Initiative (QUERI) implementation roadmap: towards sustainability of evidence-based practices in a learning health system. Med Care. 2019;57(Suppl 10 Suppl 3(10 Suppl 3)):S286–S293. doi: 10.1097/MLR.0000000000001144. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Kizer KW, Jha AK. Restoring trust in VA health care. N Engl J Med. 2014;371:295–297. doi: 10.1056/NEJMp1406852. [DOI] [PubMed] [Google Scholar]
- 11.Guthrie S, Krapels J, Lichten CA, Wooding S. 100 Metrics to Assess and Communicate the Value of Biomedical Research: An Ideas Book, Association of American Medical Colleges, RR-1606-AAMC, 2016, available at: https://www.rand.org/pubs/research_reports/RR1606.html. Accessed August 24, 2019. [PMC free article] [PubMed]
- 12.Strategic Analytics for Improvement and Learning (SAIL). Available at: https://www.va.gov/qualityofcare/measure-up/strategic_analytics_for_improvement_and_learning_sail.asp. Accessed August 25, 2019.
- 13.Meaningful Measures Hub. Available at: https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/QualityInitiativesGenInfo/MMF/General-info-Sub-Page.html. Accessed August 25, 2019.
- 14.Kirchner JE, Ritchie MJ, Pitcock JA, Parker LE, Curran GM, Fortney JC. Outcomes of a partnered facilitation strategy to implement primary care-mental health. J Gen Intern Med. 2014;29(Suppl 4(Suppl 4)):904–12. doi: 10.1007/s11606-014-3027-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Hysong SJ, Best RG, Pugh JA. Audit and feedback and clinical practice guideline adherence: making feedback actionable. Implement Sci. 2006;1:9. doi: 10.1186/1748-5908-1-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Powell BJ, Waltz TJ, Chinman MJ, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10:21. doi: 10.1186/s13012-015-0209-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Centers for Disease Control and Prevention. Description of the Science Impact Framework. Available at: https://www.cdc.gov/od/science/impact/framework.html. Accessed August 23, 2019.