Abstract
Background:
Traditional research approaches do not promote timely implementation of evidence-based innovations (EBIs) to benefit patients. Embedding research within health systems can accelerate EBI implementation by blending rigorous methods with practical considerations in real-world settings. A state-of-the-art (SOTA) conference was convened in February 2019 with five workgroups that addressed five facets of embedded research and its potential to impact healthcare. This article reports on results from the workgroup focused on how embedded research programs can be implemented into heath systems for greatest impact.
Methods:
Based on a pre-conference survey, participants indicating interest in accelerating implementation were invited to participate in the SOTA workgroup. Workgroup participants (N = 26) developed recommendations using consensus-building methods. Ideas were grouped by thematic clusters and voted on to identify top recommendations. A summary was presented to the full SOTA membership. Following the conference, the workgroup facilitators (LJD, CDH, NR) summarized workgroup findings, member-checked with workgroup members, and were used to develop recommendations.
Results:
The workgroup developed 12 recommendations to optimize impact of embedded researchers within health systems. The group highlighted the tension between “ROI vs. R01” goals—where health systems focus on achieving return on their investments (ROI) while embedded researchers focus on obtaining research funding (RO1). Recommendations are targeted to three key stakeholder groups: researchers, funders, and health systems. Consensus for an ideal foundation to support optimal embedded research is one that (1) maximizes learning; (2) aligns goals across all 3 stakeholders; and (3) implements EBIs in a consistent and timely fashion.
Conclusions:
Four cases illustrate a variety of ways that embedded research can be structured and conducted within systems, by demonstrating key embedded research values to enable collaborations with academic affiliates to generate actionable knowledge and meaningfully accelerate implementation of EBIs to benefit patients. Implications: Embedded research approaches have potential for transforming health systems and impacting patient health. Accelerating embedded research should be a focused priority for funding agencies to maximize a collective return on investment.
Keywords: Embedded research, State-of-the-art review, Partnered research
1. Introduction
The traditional research enterprise, with sponsor-funded, investigator-driven projects led by researchers employed at academic institutions, has struggled to provide relevant and timely evidence to inform health systems’ decision-making. Abundant literature1 recognizes this mismatch between health system needs, sponsor goals, and investigator research interests as contributing to a persistent, multi-year translation gap between discovery and the effective application of evidence-based innovations (EBI) that could improve health care.2
Implementation research approaches have been developed as part of the larger sphere of implementation science, which is the study of methods to promote the systematic uptake and translation of research findings and other EBI into routine practice.3 Implementation researchers aim to address these research gaps4 through “partnered”5 or “embedded research” which aims to generate knowledge about where EBIs can effectively enhance quality, safety, patient experience, access and affordability, and how to optimize EBIs for greatest benefit. There is general agreement in the literature6-11 that characteristics of embedded researchers include having strong connections within health systems, developed relationships and aligned interests across key stakeholders, including corporate executives (“C-suite”), clinical leadership, frontline operations, and patients. Embedded researchers may be affiliated with an academic institution, not-for-profit research institutions and/or employed by health systems. Embedded research has tremendous potential for applying research methods used within implementation science to generate urgently needed knowledge about how to reliably and quickly translate EBIs into routine clinical practice.11
To better understand the current state of embedded research and future directions in the U.S., Kaiser Permanente, AcademyHealth, the US Department of Veterans Affairs, the US Agency for Healthcare Research and Quality, and the Patient Centered Outcomes Research Institute convened a State-of-the-Art (SOTA) conference to explore the range of structures and roles for embedded research and to provide recommendations to key stakeholders seeking to establish robust embedded research programs. This paper describes findings and recommendations from our workgroup that focused on how to implement an embedded research program aimed at accelerating implementation of EBIs into practice.
2. Methods
2.1. Participants
The SOTA Conference was held in Pasadena, CA in February 2019 and included 116 invitees from academic institutions, funding agencies, and health systems. Invitees were asked to identify their priority interests in a pre-conference survey provided by the conference organizers. Based on this survey, attendees were divided into working groups based on expertise, experience, and interest in one of five domains: (1) Organizational Arrangements; (2) Research Support for Management Decisions; (3) Data Resources and Data Use; (4) Strengthening the Embedded Research Community; and (5) Accelerating Implementation of Embedded Research Output. This article describes output from Workgroup 5.
2.2. Workgroup process
A guiding description of embedded research was provided as a starting point for the workgroup, based on Vindrola-Padros and colleagues’ review.10 Specifically, embedded researchers typically: have an affiliation with an academic institution, develop relationships with system stakeholders, generate knowledge, and build research capacity.
The workgroup focused on how embedded researchers, often using implementation research approaches, can accelerate translation of EBIs. Pre-conference reading assignments informed participants on the nexus of embedded research and quality improvement,10 research approaches within implementation science,3 and current gaps in moving knowledge into action.12 Initial brainstorming focused on the following questions: (1) What is the range of forms and structures of embedded research commonly employed, and what organizational or contextual factors make them successful; and (2) What are the principles or guidance for learning about how to best integrate EBIs into practice quickly, effectively, and sustainably, that leverages partnerships between researchers and systems? The Workgroup used a consensus method, called the K-J Method13 or Affinity Diagram,14 often used in quality improvement, which shows the relationships between opinions, problems, solutions, and issues by placing them in related groups. It allows a broad range of ideas to be organized and prioritized. The workgroup was divided into two teams who shared prioritized ideas with the entirety of Workgroup 5. The workgroup collaboratively developed a summary that was presented to all SOTA attendees the following day. This process resulted in a set of consolidated recommendations from the workgroup. Following the conference, the workgroup facilitators (LJD, CDH, NR) wrote a summary of team findings that was member-checked15 with the workgroup and used to inform results presented here.
3. Results
Members of the workgroup came from a diverse range of roles and affiliations (Table 1). Before tackling the challenge how to implement embedded research programs in healthcare systems aimed at accelerating implementation of EBIs into practice, participants struggled with what, exactly was meant by “embedded research” and the context within which they work. To move forward, they identified a consensus vision to articulate conditions to be met for thriving, impactful embedded research that yields rapid translation of evidence to improve health care and patient outcomes. First, is that learning be maximized for everyone in the system. From the perspective of embedded researchers, the ability to attract and retain skilled research staff with the necessary data and analytical skills is an essential requisite for an embedded research program contributing to learning within the system. Barriers to retaining skilled researchers include the rise of firms focused on technology, data science including data mining and machine learning, who are drawing key talent away from health systems and research organizations. Second, alignment of goals is needed from health system executive leadership (“C-Suite”) to the frontline clinical and operations staff. A shared and aligned commitment between key health system stakeholders, who must be provided with necessary information to understand how embedded research can help meet system and patient-centered goals, and researchers, who must ensure that their work is responsive to the system’s priorities. This is especially true with the current sense of urgency among these stakeholders to disrupt current models of care. Third, health systems need robust yet timely methods for prioritizing and evaluating EBI projects. Participants asserted that embedded research programs will thrive when system leaders have incentives and the supporting infrastructure (e.g., built into performance evaluations) to sift and weigh evidence to decide on EBI investments, evaluate program implementations, and learn how best to sustain those investments through reliable, effective implementation approaches. Successful embedded research requires effective horizontal and vertical integration with open channels of partnerships, mutual measures of success, with training and learning. By strengthening and aligning stakeholder partnerships to define shared vision and priorities, participants felt that embedded research has potential to develop truly disruptive approaches to more quickly implement and appropriately de-implement innovations to advance public health and well-being.
Table 1.
Characteristic | Workgroup Members |
---|---|
Type of Affiliation | |
US Federal Governmenta | 2 |
Health Delivery System | 8 |
Academic Medical System | 3 |
Universityb | 5 |
External Entityc | 5 |
Dual Affiliationd | 3 |
Executive Leader (non-research focused)e | 9 |
Female | 16 |
Includes Department of Veterans Affairs, Centers for Disease Control.
Employed by a University academic department, may lead projects but is not embedded within an academic health system that may be affiliated with a university.
External entities include organizations offering research and/or evaluation services and a separate system network organization.
Affiliations with 2 institutions, usually university and health system or Federal Government.
Includes non-academic C-suite roles (e.g., Chief Innovation Officer, Vice President, Executive Director, etc.)
The workgroup identified the misalignment between health system priorities and research foci as the top major challenge to embedded researchers’ ability to maximize their impact. This was characterized as the tension between “ROI vs. R01,” meaning, system leaders are concerned with maximizing return on their invested dollars, whereas researchers, whether employed by a health system or academic affiliate, often rely on winning grants from funding agencies whose scientific goals and timelines often fail to align with health system leaders’ needs and priorities. Moreover, trials funded by federal programs also do not usually allow quick failures with constant revision of protocols to assess efficacy. In response, the research community is exploring tactics to accelerate and iterate in the embedded research space, including adaptive clinical trials and design thinking, both of which support “learning while doing.” As described during the SOTA, the VA’s Health Services Research and Development (HSR&D) and Quality Enhancement Research Initiative (QUERI) programs offer an approach to aligning research with system priorities that could be replicated more broadly.16,17
Ultimately, the workgroup concluded that researchers must move faster to meet the needs of health systems and the patients they serve within a dynamic political, financing, technology, and delivery landscape, and averred that traditional funding mechanisms are not yet aligned with the need for rapid, responsive research. Table 2 lists 12 recommendations for key stakeholders to develop and sustain a productive, embedded research capability. Although clearly stated, workgroup participants agreed that these recommendations are not easy to accomplish without coordinated efforts and high-level change in priorities and policies. Workgroup members asserted that all high performing systems already have embedded researchers in place to some degree. The following section describes case studies from three health systems and one multi-system research network to illustrate various manifestations of workgroup recommendations across these diverse settings. Links to specific recommendations are listed parenthetically within each case study.
Table 2.
Stakeholder Role | Recommendations |
---|---|
A. Embedded Researchers |
|
B. External Research Funders |
|
C. Health System Leaders |
|
3.1. Illustrative case studies
Case 1.
Intermountain Healthcare – Internal Research for Rapid Internal Improvements.
Like many healthcare organizations committed to quality improvement and extraordinary care, Intermountain has challenges with variation in EBI use systemwide. To develop and scale implementation competencies, embedded implementation researchers and clinicians worked with executive leadership, including quality and patient engagement leaders, to identify and deploy EBIs system-wide while studying the feasibility and effectiveness of implementation practices (C.3.). This collaborative process helps to align embedded implementation research priorities with the needs of clinicians, system leaders, and patients.18 Embedded implementation researchers are PhD-educated health services researchers trained in implementation science methods with Intermountain faculty appointments. Given their close operational alignment, these individuals are largely funded through internal operating budgets. All implementation researchers are active investigators who publish regularly with a portion of their cost recovered through internal or external grant funding mechanisms or demonstrated system improvements. External grant funding is actively sought when there is good alignment between external funding opportunities and internal priorities. For example, implementation science researchers were co-investigators on a two-year NHLBI funded grant studying methods to increase adherence to lung-protective ventilation in mechanically ventilated patients in the intensive care unit in preparation for a hybrid implementation-effectiveness trial (NCT03984175) (B.1., B.2.) Embedded researchers are part of a cross-functional co-production team, including representation from bioinformatics, systems engineering, quality improvement (C.1.) and coordination of field work with front line teams to achieve high adherence.19 Implementation decisions consider which EBIs will have the most measurable impact to ensure that system stakeholders can plan for and observe measurable improvement in patient outcomes (C.2.). An ongoing challenge to conducting embedded implementation research with necessary rigor is the reality that adaptation is necessary to increase impact and speed results (A.4.). Embedded implementation researchers use results of this work develop generalizable “playbooks” to help guide adaptation while increasing fidelity and pace of future EBI deployments and for external publication20,21 (A.3.). Early efforts have been successful with meaningful improvement in adherence in early unpublished internal studies including sustained improvements in evidence-based stroke care and improved colon bundle adherence.
Case 2.
Sanford Health – System and Research Leadership Align to Improve Care Management.
At Sanford Health,22 an embedded research infrastructure developed out of organizational commitment to innovation and top-down support encouraging rapid, rigorous and highly pragmatic research (A.4.). One of the primary keys to success has been involving embedded research leaders who lead grant-funded research portfolios with academic institutional affiliations and who are positioned at the core of health system operational structures. These embedded researchers are uniquely positioned to garner key stakeholder engagement (A.1., C.3.); ensuring research projects are meaningful to the system and have the highest potential for execution and impact on care practices. In one case, embedded research leaders strategically aligned within healthcare operations identified a need to innovate and rigorously study the implementation of a new care management tactic for patients with co-occurring chronic conditions. Given that the research leaders were already embedded at the leadership table, a team of population health researchers, with knowledge of care management EBIs were positioned to work with key operational stakeholders to develop meaningful, targeted goals specific to the system driven research initiative (A.1.). Researchers accessed administrative data to identify and prioritize a high-risk population (C.1., A.3.) and machine learning techniques were employed within a subset of clinical and claims data (N = 46,246). This data-driven focus alongside leadership support helped secure buy-in with market-led working groups, building a stakeholder-driven foundation to the project. The approach taken was highly pragmatic and flexible, a major key to success within a fast-paced, often over-taxed health system. This work was done within existing market resources; thus, research aims were aligned with the health systems’ priorities in a way that made it an obvious matter-of-fact decision to engage and fund the embedded research project while further extending research capacity and generating new knowledge of research practices and EBIs within the system. Outcome in one urban market who implemented a suite of EBIs for patients (n = 256) showed decreasing emergency department visits (53%), hospital admissions (63%), missed visits (27%), and office visits (45%). Projects like this, with clear leadership buy-in and measurable clinical outcomes, continue to garner the commitment to growing and sustaining an embedded research model throughout the organization.
Case 3.
Veterans Health Administration – Leveraging Enterprise Data to Move from Research Insights to Operational Improvements.
In the early 2000’s, a national, external audit revealed potential deficits in quality of care for acute myocardial infarction in VA relative to community care. In responding to the crisis, VA leadership quickly realized there were no reliable data on how many cardiac catheterization laboratories VA operated; how many coronary interventions VA performed; or what the outcomes were. Several years earlier, VA had established a formal embedded research program, the Quality Improvement Research Initiative (QUERI) (A.2, B.1, C.1), aligned with high-impact VA patient populations, including for ischemic heart disease. The VA national clinical leader for cardiology, Dr. Robert Jesse, served on the strategic advisory committee for the Ischemic Heart Disease QUERI, and he requested their help evaluating a national initiative to improve the quality of acute myocardial infarction care following the external audit (C.2). Dr. Jesse articulated a set of principles for the data system: data collected at the point of care (not abstracted afterwards); having the data immediately available with system-wide transparency; and deploying data collection tools that make the flow of clinical care easier, not harder (Box et al., 2010). The QUERI team helped develop and implement the Clinical Assessment Reporting and Tracking - Cath Lab (CART-CL) Program: a quality and safety organization that leveraged an internally developed electronic medical record system for cardiac catheterization procedures, integrated with the primary VA health record, that has helped the US Department of Veterans Affairs act as a learning healthcare system by using real-time data to improve clinical care.23 The CART Program has significantly evolved into a mature operational program over the last decade (C.2), and now encompasses quality programs to improve point-of-care analytics24,25 (A. 4), safety programs that review adverse events25 and reporting programs that provide transparency to facilities and operators about the performance of invasive cardiac procedures (A.5). Together, these initiatives have been associated with a significant improvement in clinical outcomes among patients undergoing invasive cardiac procedures over time, despite increasing clinical and anatomic complexity of the patients treated.26,27 Finally, CART also provided the foundation for the Ischemic Heart Disease QUERI to rigorously evaluate implementation strategies for evidence-based practices such as trans-radial coronary procedures (C.1, C.2).
Case 4.
The Health Care Systems Research Network – Infrastructure to Support Multi-Institutional Embedded Research.
The Health Care Systems Research Network (HCSRN) was founded in 1994 as the HMO Research Network by leaders from research centers embedded in 6 integrated health systems that provide both health care and insurance coverage.21 The HCSRN’s mission, generating high-quality, public domain research that improves individual and population health, is aided by a centralized infrastructure and strong collaborative ties between researchers themselves and with their parent health system leaders (C.1). The now 18-member organization, comprised of both large and small health systems, including those in rural areas, undertake collaborative research on mental health, aging, cancer, addiction, cardiovascular disease, and other topics. Three research awards spurred substantial growth of the HCSRN’s embedded research capabilities (B.1). NCI funded the Cancer Research Network22-24 from 1998 to 2018 as a population laboratory based in community-based health care systems to complement their portfolio of basic and clinical research, including funding for the HCSRN to create its Common Data Model, the Virtual Data Warehouse (VDW; B.3),25 now the cornerstone of many types of HCSRN research. A second instrumental development was a 2005 NIH contract to develop structural and operational efficiencies that accelerate the pace of research (B.2, C.1).26 Through this contract, the HCSRN streamlined approaches to administrative aspects of multisite research, including Data Use Agreements, subcontracts, and study recruitment, enabling researchers to more quickly initiate and implement large studies that may involve as many as 15 collaborating systems. Finally, an award from AHRQ spurred creation of a scalable distributed research network, which supports comparative effectiveness, safety, and utilization research (B.2)27,28. Use of a common data model and distributed research framework are technical approaches that help build inter-institutional trust while accelerating the pace of research and ensuring data security (B.3).
HCSRN’s collaborative culture enables it to conduct multicenter research that is responsive to the priorities of their parent health system.28,29 Regardless of the size of that system, smaller research centers can learn from large systems such as Kaiser Permanente, and vice versa. The HCSRN’s culture is manifested in concrete ways that other embedded researchers could replicate. Monthly grand rounds provide space to share insights and challenges in conducting embedded research (e.g., implementing pragmatic trials using EHRs; specific steps to partnering with different levels of a health system; A.2., A.4., C.4.). The HCSRN’s annual conference convenes approximately 400 attendees who share scientific results and engage in informal knowledge exchange to solve common challenges in embedded research. Scientific interest Groups (open to members and non-members) enable those with shared affinities to forge collaboration. Since many HCSRN researchers now have ≥15 years of experience conducting embedded research, the network is now focusing on training and career development for junior faculty.
4. Discussion
There is growing recognition of the potential value of embedded research, despite inconsistent definitions and sparse evidence. Workgroup recommendations provide guidance to support embedded researchers’ efforts to more quickly implement research findings to benefit patients and healthcare teams. These recommendations will not be easy to implement and will take coordinated efforts at multiple levels as listed in Table 2. As challenging as they may be to address, until they are, the potential impact of embedded researchers will fall short. Local factors, including organizational setting, size, and strategic priorities will shape how each recommendation is operationalized with individual health systems. Organizations of all sizes truly committed to supporting embedded researchers will strive to meet the recommendations leading to the workgroup’s consensus vision. Health systems do not have a strong track record in making investments needed to support this vision.30 However, many organizations already use quality improvement (QI) approaches, including Plan-Do-Study-Act (PDSA) cycles to implement change and are increasingly building on this foundation by engaging teams in continuous QI.31,32 Emerging approaches offer ways to merge principles from implementation with QI33-35; for example, PDSA cycles are at the heart of the Dynamic Sustainability Framework.36 Embedded researchers can aid systems in meeting goals related to use of QI approaches as part of a larger focus on rapid, reliable EBI implementation.
The role of embedded researchers strongly aligns with health systems’ moving toward becoming learning health systems.37,38 Two key themes common throughout the recommendations were the imperative for aligning goals39 and building close working partnerships.40 This is aligned with Bush and colleagues’ findings that when health system leaders provide impetus and research aims are aligned with those needs, the likelihood of higher benefit is quadrupled.41
Bush and colleagues41 have described specific structures and processes to support embedded research (e.g., cross-functional workgroups that partner from ideation to completion to build trust and collaboration) that is embedded within systems. Durham characterizes the importance of strong partnerships20 with health system stakeholders21 and Schmittdiel et al.42 proposes a partnership model that derives from participatory research with communities. Nevertheless, more work is needed to understand the mechanisms by which better system, patient, and clinician and staff outcomes at multiple levels, can be achieved and to justify investment in embedded research structures and teams. Though many workgroup participants came from larger integrated health systems, smaller systems can also support the work of embedded researchers. One approach is to join a network to share resources and learnings across systems as illustrated in Case 4. Another challenge for smaller systems might be funding and attracting and retaining talented researchers. However, at a smaller scale, grants, graduate student research fellowships and other options would allow these entities to maintain a smaller, yet focused program that can help accelerate implementation. Smaller practices and health systems are increasingly joining Accountable Care Organizations (ACO), which could encourage innovative research by increasing access to pooled resources and infrastructure,43-45 but the reality may not yet match its potential as a model for supporting impactful embedded research.45,46
There are several limitations related to the idea of embedded research, as conceptualized by this SOTA workgroup, and related to the recommendations. Embedded researchers have the potential to introduce bias into data collection, particularly key informant interviews. As employed researchers of the system, individuals may have vested interest in evaluation outcomes and may be perceived differently by interviewees than a researcher from an independent entity. Consideration of how external entities (e.g., contracted consultants and analysts) can support embedded research may help mitigate this. But it is more challenging for external entities to build the close partnerships and aligned goals that are key facilitating characteristics of embedded researchers. Another potential limitation is that generalizability of results from implementation studies conducted within a single system may be limited. Though conference organizers and participants explored a wide range of literature prior to the SOTA to assess and understand the role of embedded research, this review was not systematically conducted but rather relied on experience and expertise of SOTA participants. As shown in Table 1, membership of the workgroup was diverse, including key stakeholders from health systems, academic research, and external evaluation and advocacy groups; additionally, over one-third were executive-level leaders within health systems. Predominant perspectives were of relatively large integrated health systems within the U.S. Recommendations are, thus, largely based on experiences of these larger systems and may not translate to community, public health, and low- or middle-income countries outside the U.S. One member was, however, the founder and leader of a patient advocacy group; even so, patient perspectives were underrepresented, in part, because this group self-selected by indicating preference for the topic related to accelerating implementation. Other SOTA workgroups did have higher patient representation. This is a key constituency to involve as these recommendations are implemented. We note that this workgroup did not include participants from funding agencies – these stakeholders participated in a different workgroup that focused on funding and governance mechanisms. Our workgroup mentioned the need for career pathways for embedded researchers, but another SOTA workgroup focused on how to support career pathways to build capacity in embedded research.47
5. Conclusion
Embedded research models have tremendous potential for transforming health systems and having significant impact on the patients they serve. This SOTA workgroup developed recommendations to help achieve this vision by advancing embedded research structures and processes globally. By offering four diverse exemplars and concrete recommendations, we hope that other health systems may recognize how features of their system may be conducive to embedded research and undertake similar initiatives that accelerate implementation of EBIs. Realization of this vision could yield important benefits to the US healthcare system, including a better return on every research and health-system dollar invested.
Acknowledgements
In addition to the authors, we thank these “Workgroup E” members who contributed their time and ideas during the SOTA conference: Jane Brockman, MD, MSPH, Benjamin Broder, MD; Benson Hsu, MD, MBA, FAAC; Cara Lewis, PhD; George T. Blike, MD; Laura Seeff, MD; M. Rashad Massoud, MD, MPH, FACP; Maureen Fagan, DNP, MHA, FNP-BC, FAAN; Adam L. Sharp MD, MSc, and Elizabeth A McGlynn, PhD. We also thank Dr. Steven Waldo, Director of the VA CART-CL program, for help drafting the CART-CL case study.
Funding acknowledgement
Kaiser Permanente, AcademyHealth, the U.S. Department of Veterans Affairs, the US Agency for Healthcare Research and Quality, and the Patient Centered Outcomes Research Institute supported this State of the Art (SOTA) conference.
Dr. Knighton received additional support from the National Center for Advancing Translational Sciences of the National Institutes of Health (KL2TR002539).
This work was supported by the U.S. Department of Veterans Affairs, Veterans Health Administration, Health Services Research & Development Service. The views expressed are those of the authors and do not represent the views of the Department of Veterans Affairs of the US Government.
Declaration of competing interest
The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Dr. Ho declares that PMH is supported by grants from NHLBI, VA HSR&D, and University of Colorado School of Medicine. He has a research agreement with Bristol-Myers Squibb through the University of Colorado. He serves as the Deputy Editor for Circulation: Cardiovascular Quality and Outcomes.
Dr. Brock declares that Telligen has held quality improvement contracts with the Centers for Medicare & Medicaid Services, and grant funding from the Patient-Centered Outcomes Research Institute.
The remaining authors have nothing to disclose.
References
- 1.Whicher D, Rosengren K, Siddiqi S, Simpson L, eds. The Future of Health Services Research: Advancing Health Systems Research and Practice in the United States. Washington, DC: National Academy of Medicine; 2018. [PubMed] [Google Scholar]
- 2.Balas EA, Boren SA. Managing clinical knowledge for health care improvement. Yearb Med Inform. 2000;(1):65–70. [PubMed] [Google Scholar]
- 3.Eccles M, Mittman B. Welcome to implementation science. Implement Sci. 2006;1(1): 1. [Google Scholar]
- 4.Bauer M, Damschroder L, Hagedorn H, Smith J, Kilbourne A. An introduction to implementation science for the non-specialist. BMC Psychology. 2015;3(1):32. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Glasgow R, Green L, Taylor M, Stange K. An evidence integration triangle for aligning science with policy and practice. Am J Prev Med. 2012;42(6):646–654. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Churruca K, Ludlow K, Taylor N, Long JC, Best S, Braithwaite J. The time has come: embedded implementation research for health care improvement. J Eval Clin Pract. 2019;25(3):373–380. [DOI] [PubMed] [Google Scholar]
- 7.Forrest CB, Chesley FD Jr, Tregear ML, Mistry KB. Development of the learning health system researcher core competencies. Health Serv Res. 2018;53(4):2615–2632. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Psek WA, Stametz RA, Bailey-Davis LD, et al. Operationalizing the learning health care system in an integrated delivery system. EGEMS (Wash DC). 2015;3(1):1122. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Thompson EE, Steiner JF. Embedded research to improve health: the 20th annual HMO Research Network conference, March 31-April 3, 2014, Phoenix, Arizona. Clin Med Res. 2014;12(1-2):73–76. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Vindrola-Padros C, Pape T, Utley M, Fulop NJ. The role of embedded research in quality improvement: a narrative review. BMJ Qual Saf. 2017;26(l):70–80. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Wolfenden L, Yoong SL, Williams CM, et al. Embedding researchers in health service organizations improves research translation and health service performance: the Australian Hunter New England Population Health example. J Clin Epidemiol. 2017; 85:3–11. [DOI] [PubMed] [Google Scholar]
- 12.Kreindler S What if implementation is not the problem? Exploring he missing links between knowledge and action. Int J Health Plann Mgmt. 2016;31:208–226. November. [DOI] [PubMed] [Google Scholar]
- 13.Spool JM. The KJ-technique: a group process for establishing priorities. https://articles.uie.com/kj_technique/; 2004. Accessed January 25, 2019.
- 14.Institute for Healthcare Improvement. Idea generation tools: brainstorming, affinity grouping, and multivoting. http://www.ihi.org/resources/Pages/Tools/BrainstormingAffinityGroupingandMultivoting.aspx; 2019. Accessed January 30, 2019. [Google Scholar]
- 15.Birt L, Scott S, Cavers D, Campbell C, Walter F. Member checking: a tool to enhance trustworthiness or merely a nod to validation? Qual Health Res. 2016;26(13): 1802–1811. [DOI] [PubMed] [Google Scholar]
- 16.Kilbourne AM, Elwy AR, Sales AE, Atkins D. Accelerating research impact in a learning health care system: VA’s quality enhancement research initiative in the choice act era. Med Care. 2017;55(Suppl 7 Suppl 1):S4–S12. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Kilbourne AM, Goodrich DE, Miake-Lye I, Braganza MZ, Bowersox NW. Quality enhancement research initiative implementation roadmap: toward sustainability of evidence-based practices in a learning health system. Med Care. 2019;57(Suppl 10 Suppl 3):S286–S293. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Knighton AJ, Grissom CK, Knowlton KU, et al. How community-based health systems can embrace research in the fee-for-value era. Qual Manag Health Care. 2018;27(3): 180–182. [DOI] [PubMed] [Google Scholar]
- 19.Knighton AJ, McLaughlin M, Blackburn R, et al. Increasing adherence to evidence-based clinical practice. Qual Manag Health Care. 2019;28(1):65–67. [DOI] [PubMed] [Google Scholar]
- 20.Allen L, Knighton AJ, Wolfe D, et al. Implementing evidence-based clinical practice in the critical care setting. Qual Manag Health Care. 2020;29(2): 123–125. [DOI] [PubMed] [Google Scholar]
- 21.Stenehjem E, Wallin A, Fleming-Dutra KE, et al. Antibiotic prescribing variability in a large urgent care network: a new target for outpatient stewardship. Clin Infect Dis. 2020;70(8): 1781–1787. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Sanford health, https://www.sanfordhealth.org/; 2019. Accessed October 2, 2019.
- 23.Maddox TM, Plomondon ME, Petrich M, et al. A national clinical quality program for Veterans Affairs catheterization laboratories (from the Veterans Affairs clinical assessment, reporting, and tracking program). Am J Cardiol. 2014;114(11): 1750–1757. [DOI] [PubMed] [Google Scholar]
- 24.Valle JA, Glorioso TJ, Bricker R, et al. Association of coronary anatomical complexity with clinical outcomes after percutaneous or surgical revascularization in the Veterans Affairs clinical assessment reporting and tracking program. JAMA Cardiol. 2019;4(8):727–735. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Doll JA, Plomondon ME, Waldo SW. Characteristics of the quality improvement content of cardiac catheterization peer reviews in the Veterans Affairs clinical assessment, reporting, and tracking program. JAMA Netw Open. 2019;2(8), e198393. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Waldo SW, Gokhale M, O’Donnell CI, et al. Temporal trends in coronary angiography and percutaneous coronary intervention: insights from the VA clinical assessment, reporting, and tracking program. JACC Cardiovasc Interv. 2018;11(9):879–888. [DOI] [PubMed] [Google Scholar]
- 27.Hall PS, O’Donnell CI, Mathew V, et al. Outcomes of Veterans undergoing TAVR within Veterans Affairs medical centers: insights from the Veterans Affairs clinical assessment, reporting, and tracking program. JACC Cardiovasc Interv. 2019;12(21): 2186–2194. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Simon GE, Johnson E, Lawrence JM, et al. Predicting suicide attempts and suicide deaths following outpatient visits using electronic health records. Am J Psychiatr. 2018;175(10):951–960. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Sperl-Hillen JM, Rossom RC, Kharbanda EO, et al. Priorities wizard: multisite web-based primary care clinical decision support improved chronic care outcomes with high use rates and high clinician satisfaction rates. EGEMS (Wash DC). 2019;7(1):9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Reed JE, Card AJ. The problem with plan-do-study-act cycles. BMJ Qual Saf. 2016;25 (3):147–152. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Swensen SJ, Dilling JA, Mc Carty PM, Bolton JW, Harper CM Jr. The business case for health-care quality improvement. J Patient Saf. 2013;9(1):44–52. [DOI] [PubMed] [Google Scholar]
- 32.Baron AN, Hemler JR, Sweeney SM, et al. Effects of practice turnover on primary care quality improvement implementation. Am J Med Qual. 2020;35(1):16–22. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Koczwara B, Stover AM, Davies L, et al. Harnessing the synergy between improvement science and implementation science in cancer: a call to action. J Oncol Pract. 2018;14(6):335–340. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Wagner EH, Glasgow RE, Davis C, et al. Quality improvement in chronic illness care: a collaborative approach. Jt Comm J Qual Improv. 2001;27(2):63–80. [DOI] [PubMed] [Google Scholar]
- 35.Horwitz LI, Kuznetsova M, Jones SA. Creating a learning health system through rapid-cycle, randomized testing. N Engl J Med. 2019;381(12):1175. [DOI] [PubMed] [Google Scholar]
- 36.Chambers DA, Glasgow RE, Stange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. 2013;8 (1):117. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Greene SM, Reid RJ, Larson EB. Implementing the learning health system: from concept to action. Ann Intern Med. 2012;157(3):207–210. [DOI] [PubMed] [Google Scholar]
- 38.Guise JM, Savitz LA, Friedman CP. Mind the gap: putting evidence into practice in the era of learning health systems. J Gen Intern Med. 2018;33(12):2237–2239. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Psek W, Davis FD, Gerrity G, et al. Leadership perspectives on operationalizing the learning health care system in an integrated delivery system. EGEMS (Wash DC). 2016;4(3):1233. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Hamzeh J, Pluye P, Bush PL, Ruchon C, Vedel I, Hudon C. Towards an assessment for organizational participatory research health partnerships: a systematic mixed studies review with framework synthesis. Eval Progr Plann. 2019;73:116–128. [DOI] [PubMed] [Google Scholar]
- 41.Bush PL, Pluye P, Loignon C, et al. Organizational participatory research: a systematic mixed studies review exposing its extra benefits and the key factors associated with them. Implement Sci. 2017;12(1):119. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Schmittdiel JA, Grumbach K, Selby JV. System-based participatory research in health care: an approach for sustainable translational research and quality improvement. Ann Fam Med. 2010;8(3):256–259. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.McMurry AJ, Zhang R, Foxman A, Reiter L, Schnel R, Brandman D. Using Routinely Collected Patient Data to Support Clinical Trials Research in Accountable Care Organizations. arXiv; 2018. preprint arXiv: 180700668. [Google Scholar]
- 44.Chukmaitov AS, Harless DW, Bazzoli GJ, Deng Y. Factors associated with hospital participation in centers for medicare and medicaid services’ accountable care organization programs. Health Care Manag Rev. 2019;44(2): 104–114. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Perrin JM, Zimmerman E, Hertz A, Johnson T, Merrill T, Smith D. Pediatric accountable care organizations: insight from early adopters. Pediatrics. 2017;139(2), e20161840. [DOI] [PubMed] [Google Scholar]
- 46.Kim D, Funk RJ, Yan P, Nallamothu BK, Zaheer A, Hollingsworth JM. Informal clinical integration in medicare accountable care organizations and mortality following coronary artery bypass graft surgery. Med Care. 2019;57(3):194–201. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Yano EM, Gluck M, Resnick A, Mistry K, Kwon H. Fostering Career Pathways in Embedded Research: Training Needs, Supporting Career Development, and Accelerating Impactful Career Trajectories under Review.
- 48.Lukas CV, Holmes SK, Cohen AB, et al. Transformational change in health care systems: an organizational model. Health Care Manag Rev. 2007;32(4):309–320. [DOI] [PubMed] [Google Scholar]
- 49.Kontoghiorghes C, Awbre SM, Feurig PL. Examining the relationship between learning organization characteristics and change adaptation, innovation, and organizational performance. Hum Resour Dev Q. 2005; 16(2): 185–212. [Google Scholar]
- 50.Scheirer MA, Dearing JW. An agenda for research on the sustainability of public health programs. Am J Publ Health. 2011;101(11):2059–2067. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.Shediac-Rizkallah MC, Bone LR. Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice and policy. Health Educ Res. 1998;13(1):87–108. [DOI] [PubMed] [Google Scholar]
- 52.Glasgow RE, Chambers D. Developing robust, sustainable, implementation systems using rigorous, rapid and relevant science. Clin Transl Sci. 2012;5(1):48–55. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 53.Riley WT, Glasgow RE, Etheredge L, Abernethy AP. Rapid, responsive, relevant (R3) research: a call for a rapid learning health research enterprise. Clin Transl Med. 2013; 2(1):10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 54.Stoto M, Parry G, Savitz L. Analytical methods for a learning health system: 4. Delivery system science. EGEMS (Wash DC). 2017;5(1):31. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Hill JE, Stephani AM, Sapple P, Clegg AJ. The effectiveness of continuous quality improvement for developing professional practice and improving health care outcomes: a systematic review. Implement Sci. 2020;15(1):23. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 56.Platt JE, Raj M, Wienroth M. An analysis of the learning health system in its first decade in practice: scoping review. J Med Internet Res. 2020;22(3), e17026. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 57.Williams NJ, Wolk CB, Becker-Haimes EM, Beidas RS. Testing a theory of strategic implementation leadership, implementation climate, and clinicians’ use of evidence-based practice: a 5-year panel analysis. Implement Sci. 2020; 15(1): 10. [DOI] [PMC free article] [PubMed] [Google Scholar]