How Dissemination and Implementation Science Can Contribute to the Advancement of Learning Health Systems

Abstract

Many health systems are working to become learning health systems (LHSs), which aim to improve the value of health care by rapidly, continuously generating evidence to apply to practice. However, challenges remain to advance toward the aspirational goal of becoming a fully mature LHS. While some important challenges have been well-described (i.e., building system-level supporting infrastructure and the accessibility of inclusive, integrated, and actionable data), other key challenges are underrecognized, including balancing evaluation rapidity with rigor, applying principles of health equity and classic ethics, focusing on external validity and reproducibility (generalizability), and designing for sustainability. Many LHSs focus on continuous learning cycles, but with limited consideration of issues related to the rapidity of these learning cycles, as well as the sustainability or generalizability of solutions. Some types of data have been consistently underrepresented, including patient-reported outcomes and preferences, social determinants, and behavioral and environmental data; the absence of which can exacerbate health disparities. A promising approach to addressing many challenges that LHSs face may be found in dissemination and implementation (D&I) science. With an emphasis on multilevel dynamic contextual factors, representation of implementation partner engagement, pragmatic research, sustainability, and generalizability, D&I science methods can assist in overcoming many of the challenges facing LHSs. In this article, the authors describe the current state of LHSs and challenges to becoming a mature LHS; propose solutions to current challenges, focusing on the contributions of D&I science with other methods; and propose key components and characteristics of a mature LHS model that others can use to plan and develop their LHSs.

As health care strives to meet the Quadruple Aim of improved outcomes and experiences for patients and health care teams and lower costs,¹ it is clear that changes in the health care delivery system are needed. In this milieu, the concept of a learning health system (LHS) was introduced in 2007² and serves as an aspirational goal for many health systems in which “… the best evidence for the collaborative healthcare choices of each patient and provider; … drive the process of discovery as a natural outgrowth of patient care; and ensure innovation, quality, safety, and value in healthcare.”³ Foundational elements of an LHS are rapid, continuous learning cycles to generate and apply evidence to practice to improve the value of health care. Balancing the demands of research and clinical practice is no small feat and attaining the status of a fully mature LHS that harmoniously blends evaluation and clinical care is an ever-moving target.⁴ Informed by the National Academy of Medicine’s³ and other experts’ work, the Lannon et al maturity grid can be used to evaluate and guide progress toward the aspirational goal of becoming a fully mature LHS.⁵ Common challenges to advancing along this maturity grid include lack of shared mission across research and clinical enterprises, limited informatics resources (e.g., expertise, data access, data quality), insufficient engagement with implementation partners, and lack of support from internal leadership and policy makers.^6-10

These well-known challenges are typically discussed separately and the scope of proposed solutions is usually limited. A complementary and promising approach to address these challenges more holistically, and through an equity lens, may be found in the field of dissemination and implementation (D&I) science. D&I science is the study of how evidence-based practices are feasibly adopted, implemented, and sustained in real-world settings.^11,12 It includes the study of how to best communicate with potential implementers to enact widespread uptake and effective use of practices. As such, D&I science has been recognized as highly relevant to increase the speed of translating evidence into practice^13,14 and thus can assist health systems in advancing toward becoming mature LHSs. With an emphasis on multilevel dynamic contextual factors, representation of implementation partner engagement, pragmatic research, sustainability, and generalizability, D&I science methods can assist in overcoming many of the challenges facing LHSs.

Therefore, the purpose of this article is to describe the current state of LHSs and challenges to becoming a mature LHS; propose solutions to current challenges, focusing on the contributions of D&I science with other methods, including health equity-focused, multilevel implementation partner engagement and pragmatic research; and propose key components and characteristics of a mature LHS model that others can use to plan and develop their LHSs. To this end, we conducted an environmental scan, which included consultations with experts and a literature review of key opinion pieces, reviews, and articles highlighting LHS examples (see Supplemental Digital Appendix 1 at http://links.lww.com/ACADMED/B304 for details).

The Current State of LHSs: An Environmental Scan

We conducted our environmental scan (see above) in 2022 to further elucidate the characteristics of LHSs, identify current challenges LHSs face, and inform potential solutions to these challenges. In doing so, we identified several organizations that have made significant progress toward achieving the goal of becoming a mature LHS at the organizational, state, and/or national level.¹⁵

The LHS examples we identified can be characterized in terms of their governance structure for prioritizing issues to study via learning cycles. As shown by the examples highlighted in Table 1, some LHSs include a centralized, formal governance structure and standard procedures for prioritizing issues to guide learning cycles,^10,16 while others are more decentralized.^17-21 Decentralized LHSs, such as PEDSnet,^17,18 are often data networks that can be readily analyzed by other systems in a “bring your own researcher and/or analytics” approach; however, the data they provide may not be fully relevant to another health system’s context. In contrast, centralized LHSs often have the expertise to conduct learning cycles but may have limited access to quality and actionable data to generate meaningful information.¹⁹ Some LHSs also function primarily as consultancy services, often housed within academic departments^22,23 but are limited if they are not a standard part of a health system’s operations. Irrespective of format, existing LHSs consistently emphasize the extensive time and effort required to establish an initial LHS infrastructure.

Table 1.

Comparison of the Strengths and Challenges of Select LHS Examples^a

LHS	Level and description	Strengths	Challenges to becoming a mature LHS
PEDSnet17,18	- National level - Decentralized clinical research network of children’s hospitals - Includes EHR data and patient engagement infrastructure to support retrospective and prospective learning cycles	Accessibility of inclusive, integrated, and actionable data: - Robust access to actionable data - Availability of cross-institutional data	Designing for sustainability: - Separate funding required to use resources - Challenging to sustain learning cycles and partner engagement after funding ends Focusing on external validity and reproducibility (generalizability): - No standardized approach to prioritize learning cycle generalizability Building system-level supporting infrastructure: - Relies on local expertise to analyze data and implement change Accessibility of inclusive, integrated, and actionable data: - Limited to pediatric EHR data
Vanderbilt16	- Regional level - Centralized LHS infrastructure with formal governance, standardized processes, and transdisciplinary expertise to address local health system issues	Designing for sustainability: - Currently funded by organization (originally grant funded) Focusing on external validity and reproducibility (generalizability): - Explicitly stated as a priority Building system-level supporting infrastructure: - Capacity building training Other: - Prioritizes multilevel implementation partner engagement	Building system-level supporting infrastructure: - Only conducts learning cycles when randomized designs are possible and can be completed in 1–2 years - Limited to prioritized issues; resources not available to address all issues
Connected Health Cities19	- Regional level - Centralized LHS funded at national level to address regional health inequities by analyzing routinely captured health information and feeding results back to clinical staff	Balancing evaluation rapidity with rigor: - Addresses issues based on each local region’s unique context Building system-level supporting infrastructure: - Strong support at leadership and policy level Other: - Multilevel implementation partner engagement is prioritized, with an emphasis on patients, the community, and policy makers	Designing for sustainability: - Funding beyond initial investment unclear - High rates of staff turnover Focusing on external validity and reproducibility (generalizability): - Generalizability of learning cycles across regions is unclear Building system-level supporting infrastructure: - Inefficiencies due to operating LHS independently in each region - Limited capacity building training - Challenges in acquiring methodologic and analytic expertise Accessibility of inclusive, integrated, and actionable data: - Challenges in accessing quality and actionable data Other: - Not all learning cycles are formally evaluated
University of Michigan22	- National and local level - Centralized, dedicated consultative service housed in an academic department with transdisciplinary expertise and collaborations	Building system-level supporting infrastructure: - Capacity building training and consultation within and external to the institution - Diverse methodologic and analytic expertise, including D&I science expertise	Designing for sustainability: - Continuation of learning cycles is dependent on local priorities Building system-level supporting infrastructure: - Not integrated as a standard process but as an optional consultative service Accessibility of inclusive, integrated, and actionable data: - Quality and actionable data dependent on local health system
DARTNet92	- National level - Decentralized clinical research network including cross-institutional data and partnerships that allow sharing of personnel resources to support retrospective and prospective learning cycles	Designing for sustainability and building system-level supporting infrastructure: - Uses a viable business model - Facilitates sharing of methodologic and analytic expertise Accessibility of inclusive, integrated, and actionable data: - Availability of cross-institutional data Other: - Allows for benchmarking between practices	Designing for sustainability: - Continuation of learning cycles is dependent on local priorities - Institutions must opt in and pay to play to use resources Focusing on external validity and reproducibility (generalizability): - Generalizability of learning cycles is dependent on local priorities Accessibility of inclusive, integrated, and actionable data: - Quality and actionable data dependent on local health system Other: - Integration of implementation partner engagement is unclear

Abbreviations: LHS, learning health system; EHR, electronic health record; D&I, dissemination and implementation.

^aThe LHS examples presented here were identified in an environmental scan performed by the authors in 2022. See the main text and Supplemental Digital Appendix 1 at [LWW INSERT LINK] for more information on the environmental scan. The examples selected for this table were chosen to illustrate the different types of LHSs and their characteristics. There are other examples of LHSs that could have been selected, some of which are mentioned elsewhere in this article. Across all LHS examples evaluated in the environmental scan, there was a lack of discussion on the following issues: balancing evaluation rapidity with rigor, applying principles of health equity and classic ethics, focusing on generalizability and designing for sustainability.

Based on our environmental scan, there is a general consensus regarding the importance of a systems thinking approach—in which there is engagement and support from patients, community members, clinicians, researchers, leaders, and policy makers for the overarching LHS and learning cycles conducted.^4,24,25 However, the level, depth, and longitudinal attention to systems issues and inclusion of representative invested implementation partners is variable and methods to align learning cycle solutions with multilevel contextual factors are infrequently reported. Engagement and integration of higher levels of the socio-ecologic spectrum²⁶ (e.g., policy) is also less common, but is a unique strength of the Swiss LHS.²⁴

Many LHSs point to the challenges of sustainability over time and generalizability of findings. LHSs often report highly variable, short-term funding as a result of national reimbursement models that do not currently incentivize learning. To support a sustainable culture of rapid learning and an LHS-capable workforce, some LHSs report a focus on training.^16,25 There are also some national efforts to increase the number of formal LHS training programs.^27,28 With respect to generalizability, many LHSs describe approaches to conducting learning cycles with limited consideration of how the findings can be applied to other health systems.¹⁹ The Vanderbilt LHS is one example that prioritizes external generalizability.¹⁶

There are a few published examples of LHSs incorporating D&I science to address the challenges LHSs face.^22,25,29-31 Such examples are limited to using D&I science to plan and evaluate LHSs that focus on addressing a specific public health issue,³⁰ evaluate (not plan or iterate) learning cycles that address diverse system issues,³¹ and understand the interactions of the multilevel contextual factors that can generally influence an LHS.¹⁵ For example, Alliances to Disseminate Addiction Prevention and Treatment is applying D&I science to plan and evaluate their LHS that specifically aims to connect at-risk youth in the justice system of 8 Indiana counties with mental health services.³⁰ Some health systems indicate plans to apply D&I science to establish²⁵ or further develop¹⁶ their LHSs. Others have described the advantages of using D&I science to integrate precision health into LHSs³² or promote the rapid uptake of evidence into practice.³³ Veterans Affairs has also created and iteratively applied the Quality Enhancement Research Initiative Implementation Roadmap, which can be used to systematically apply D&I science to learning cycles.³⁴ However, integrating D&I science with LHSs remains underutilized in published examples of LHSs and its value is understated in key opinion pieces, literature reviews, and position statements. There has been, to our knowledge, no focused discussion of how D&I science can be applied to address ongoing challenges to developing mature LHSs.

Table 1 provides examples of LHSs that illustrate a spectrum of maturity and highlights both strengths and challenges of each. In conducting our environmental scan, we identified challenges previously described by others related to building system-level supporting infrastructure and the accessibility of inclusive, integrated, and actionable data. We also identified 4 other challenges LHSs face: balancing evaluation rapidity with rigor, applying principles of health equity and classic ethics (e.g., do no harm), focusing on external validity and reproducibility (generalizability), and designing for sustainability. To our knowledge, there has been limited attention to the issue of health equity in LHSs^35,36 and a lack of discussion surrounding the issues of balancing evaluation rapidity with rigor, designing for sustainability, and focusing on generalizability. See Supplemental Digital Appendix 2 (at http://links.lww.com/ACADMED/B304) for a glossary of key terms used throughout this article.

Advancing LHSs

In Figure 1, we outline our view of the key components and characteristics of an aspirational, mature LHS that emphasizes D&I science to directly address the challenges we identified in our environmental scan. This mature LHS model includes iterative and rapid learning cycles with implementation partner engagement and support across all levels of the socio-ecologic spectrum (e.g., local health system unit and organization, community, public policy) and emphasizes the integration of principles, concepts, and methods from D&I science, informatics, inclusive approaches to precision health, health equity-focused and multilevel implementation partner engagement, quality improvement (QI), and pragmatic research. In this model, the LHS includes an infrastructure at the public policy and organizational (health system) levels that provides both financial support and a shared culture of embedded learning with partners across multiple community sectors (i.e., multisector collaborations). Unique to this model is the intersection of D&I science with the other methods mentioned above to address challenges facing LHSs.

Figure 1.

An aspirational model for a mature LHS. Positive outcomes of a fully mature LHS include high value of health care (high-quality, cost-effective, timely, patient-centered), health equity, generalizability, sustainability, and scalability of learning cycle findings. Application of this model addresses the following challenges facing existing LHSs: building system-level supporting infrastructure; the accessibility of inclusive, integrated, and actionable data; balancing evaluation rapidity with rigor; applying principles of health equity and classic ethics; focusing on external validity and reproducibility (generalizability); and designing for sustainability. Abbreviations: LHS, learning health system; D&I, dissemination and implementation.

^aFew examples of LHSs at this level.

This model integrates concepts from pragmatic research and the social ecological model of health with prior conceptual models of the LHS. In the model’s learning cycles, Greene et al’s³⁷ classic LHS model is merged with a traditional QI plan, do, study, act cycle. Each learning cycle includes input from representative internal and external partners to adapt in ways that promote characteristics of sustainability, generalizability, and scalability. In the center of the learning cycles are key components that should be used together, including D&I science. The model is depicted as a Venn diagram, similar to the classic social ecological model, to highlight how learning cycles can be conducted at multiple levels (i.e., unit, organizational, community, and public policy) and that a mature LHS requires support and engagement of partners across multiple levels to achieve this robust infrastructure (e.g., financial support, organizational and community commitment, inclusive access to actionable data). The model recognizes that learning cycles are inherently complex and require a systems thinking approach. As shown, LHSs operating at higher levels of the socio-ecologic spectrum run the risk of enacting change at a slower pace, but they have potential for broader, and thus more sustainable, population health impacts. This model also specifies the positive outcomes and health impacts of a mature LHS: high value of health care (high-quality, cost-effective, timely, patient-centered), health equity, generalizability, sustainability, and scalability of learning cycle findings. While the goal of attaining high-value health care is regularly discussed, the other outcomes are infrequently discussed.

In subsequent sections, we describe the 4 underrecognized challenges LHSs face that we identified in our environmental scan, along with potential solutions or key components (see Table 2) from our mature LHS model (Figure 1). D&I science methods can also assist in addressing the well-recognized challenges LHSs face (building public policy-, community-, and organizational-level supporting infrastructure and the accessibility of inclusive, integrated, and actionable data); however, we focus our discussion on the underrecognized, but arguably equally important, challenges.

Table 2.

Key Components and Characteristics for an Aspirational, Fully Mature LHS^a

Key component	Characteristics
D&I science	Prioritizes pragmatic research methods to consider external validity and increase the rapidity of translating evidence into practice Provides several theories, models, and frameworks that: guide comprehensive and systematic consideration of contextual issues and emphasize multilevel implementation partner engagement; thereby, facilitating sustainability, generalizability, equity, and scalability of learning cycle findings inform strategies to test, implement (or de-implement), and adapt learning cycles guide pragmatic outcome evaluations (that consider the 5 R’s [see below]) Facilitates effective dissemination of learning cycles internally (scale up) and externally (scale out) to other units or organizations
Informatics	Provides the data infrastructure needed to transform actionable data into information that can be meaningfully applied and used in learning cycles, via: software and technology analytic tools and expertise standards and ontologies
Inclusive approaches to precision health	Comprehensively integrates and applies holistic, diverse types of data, including environmental, biological, clinical (e.g., EHR, medical or pharmacy claims), social determinants, and economic data, to provide high-value, individualized patient care Comprehensive integration of data is necessary to avoid exacerbation of health disparities
Health equity-focused and multilevel implementation partner engagement	Supports a systems thinking approach and alignment of learning cycles with the shared priorities, values, and needs of multiple types of partners across the socio-ecologic spectrum26 Includes representation of those from marginalized and outlier populations Encourages multilevel investment in optimizing the culture of learning in which: reimbursement mechanisms incentivize learning capacity building training develops a workforce with the requisite skills and mindset for learning transdisciplinary researchers and clinicians are well integrated into implementation teams infrastructure and regulatory processes are established that promote rapid, embedded research and maintain privacy and ethical standards potential unintended consequences are anticipated and mitigated
Quality improvement	Includes iterative methods and culture to improve the value of health care delivery at the local (i.e., unit or organization) level
Pragmatic research	Includes practical but robust methods to improve the value of health care in a manner that is generalizable to clinical practice14 and prioritizes the 5 R’s:44 relevance to partners rapidity and recursiveness of findings rigor in real-world situations replicability by others resources required are reported

Abbreviations: LHS, learning health system; D&I, dissemination and implementation; EHR, electronic health record.

^aThe key components were identified in an environmental scan performed by the authors in 2022. See the main text and Supplemental Digital Appendix 1 at [LWW INSERT LINK] for more information on the environmental scan. Several of the characteristics in the right-hand column are shared by multiple key components in the left-hand column but are only noted for one key component in this table for explanatory purposes and simplification.

Challenge #1: Balancing evaluation rapidity with rigor

One major challenge for an LHS is how to improve health care through rapid generation and application of evidence that is also rigorous and trustworthy. Historically, generation of research has been slow and translation to practice even slower.³⁸

Pragmatic research solutions.

LHSs should capitalize on novel pragmatic study designs and enrollment methods that can expedite the generation of high-quality evidence, while allowing for the practical and ethical conduct of research embedded within routine clinical settings.³⁹ The stepped wedge trial design is one pragmatic trial design that allows each group to randomly cross over from a control to an intervention at different times. When there is equipoise between multiple possible interventions, other forms of adaptive trial designs⁴⁰ or multi-arm A/B trial designs can serve to robustly and more expeditiously evaluate differences compared to traditional, sequential trial designs. Opportunities to learn from natural experiments should also be embraced.⁴¹

Tools such as the Pragmatic Explanatory Continuum Indicator Summary (PRECIS)-2 can guide design decisions and evaluation, including ways to develop and test procedures or processes that can be employed under usual care conditions with limited resources, time, and modest expertise.^42,43 The 5 R’s model provides a pragmatic framework to consider rapidity and rigor of learning cycles and informs research that is relevant to partners, rapid and recursive (iterative), rigorous, and replicable and that reports on the resources required.⁴⁴

Efficient pragmatic trial designs need to be paired with more efficient means of enrolling and consenting participants that can be done feasibly within the context of routine patient care.⁴⁵ To support real-world implementation and effectiveness research, institutional review boards are increasingly supporting waivers of informed consent for research that aligns with the Quadruple Aim.⁴⁰ Pragmatic approaches to consent subjects (e.g., waiver, opt out) are critical for LHSs to achieve the symbiotic goal of practice-based research and research-based practice and must align with publication requirements.

Informatics solutions.

Big data coupled with artificial intelligence and clinical decision support (CDS) systems can also increase learning cycle rapidity.⁴⁶ Artificial intelligence can expedite the generation and translation of evidence into practice by identifying problems without a priori hypotheses and automating CDS systems. Well-designed CDS can expedite translation of evidence into practice by surfacing digestible and actionable recommendations at the right time (e.g., alerts, reminders). The COVID-19 pandemic has led to greater integration and acceptance of informatics into routine patient care settings and taught health care providers that they can move faster.⁴⁷

Although not limited to the field of informatics, agile design processes are often used to increase rapidity and, thus, could increase learning cycle rapidity.^48,49 Agile tenets, such as the Scaled Agile Framework, that emphasize micro-iterative processes could be used to augment the speed of learning cycles.⁵⁰ This approach prioritizes efficiency and creating a first version of a solution that meets the minimum criteria to be useful (minimum viable product), not a perfect first version.⁵¹ The related adage of “perfect is the enemy of good,”⁵² which is a major culture shift for health care, enables rapid learning.

Challenge #2: Applying principles of health equity and classic ethics

As with any innovation, as LHSs mature they have the potential to introduce or reinforce negative sequelae. Aligned with classic ethics principles, LHSs need to practice the principle of “do no harm” by anticipating (and sometimes proactively modeling potential downstream consequences) and mitigating potential downstream consequences and proactively promoting health equity.³⁵ Health equity should be thought of as both a process and outcome. While an ideal outcome of LHSs is improved health equity^35,36 and there are examples of LHSs created to address targeted health disparities,³⁶ LHSs also have the potential to exacerbate health disparities.

Inclusive approaches to precision health solutions.

Aligned with the goals of precision health, mature LHSs comprehensively integrate diverse data sources to consider holistic issues that impact patient care; however, these goals have not yet been realized. Social determinants of health are not always captured in electronic health records (EHRs) and LHSs that rely solely on EHR data are unable to address these determinants.^53,54 LHSs need to aspire to the goals of precision health and consider all sources of data that impact patient care. LHSs should collect and integrate data from traditional (e.g., EHR) and non-traditional sources (e.g., wearables, digital apps) that include behavioral and environmental data.⁵⁵ With the push for precision health and as new data sources, such as omics data, become available, LHSs need to be cautious and avoid the pitfall of chasing the novel and exciting at the expense of other data (e.g., social determinants, health behaviors, patient preferences) that have clear impacts on health care quality and equity.⁵⁶

Health equity-focused implementation partner engagement and D&I science solutions.

Other potential instigators of health disparities are selection bias in the implementation partners engaged which leads to insufficient representativeness of important perspectives in learning cycles. The types of invested implementation partners engaged should include those directly or indirectly impacted (e.g., patients, clinicians, the community, public policy) and should represent not just the average perspective, but the perspectives of outliers, including patients from groups that have historically been marginalized.^35,57 Invested implementation partners should also contribute to the identification of learning cycle outcome measures, which should include measures to proactively evaluate the representativeness of and equity among the populations impacted. The Reach, Effectiveness, Adoption, Implementation, Maintenance model (RE-AIM) is one of the most commonly cited D&I science frameworks⁵⁸ and can be used to evaluate and promote the health equity of learning cycles.^40,58-61

Informatics solutions.

Artificial intelligence (e.g., machine learning predictions) learns from existing patterns and when it learns from patterns of human behavior that include inherent biases, it can reinforce health inequities.⁶² Additionally, clinical predictions that do not account for factors such as social determinants of health can lead to inaccurate prognostic estimates and under- or overtreatment.⁵⁴ Clinical prediction equations should be used but with consideration of markers of social disadvantage and health determinants to avoid further marginalization of minorities.

Other solutions.

As LHSs mature and there is greater synergy between research and practice, there are also increasing challenges surrounding classic ethics principles of respect for persons and their confidentiality. These issues have been described at length by others^36,40,45,63 and are only briefly reiterated here for completeness. To mitigate these issues, LHSs need to be accountable and transparent in their decisions with patients.³⁶ Oversight from ethical and regulatory bodies with input from patients, families, and other implementation partners is also critical.^36,63

Challenge #3: Focusing on external validity and reproducibility (generalizability)

External validity and reproducibility of LHS activities is another challenge that is infrequently prioritized but that is needed for broad translation of evidence into practice. Replication is enhanced by integrating pragmatic research methods¹⁴ and transparently reporting learning cycle findings with a description of the contextual characteristics of the setting. Reproducibility requires flexibility in how a solution or intervention is executed (local adaptation) in different settings, while also preserving the core components or functions of the intervention (fidelity).⁶⁴ Adaptations should be expected when solutions are used in actual real-world settings as these settings are often unpredictable and highly heterogeneous. When scaling up within an institution or scaling out to other institutions, a solution will inevitably require some adaptation to fit the unique contextual factors of the new setting.⁶⁵

D&I science solutions.

Using D&I science methods increases the likelihood that evidence will be translated into practice.¹³ Theories, models, and frameworks (TMFs) are cornerstones of D&I science and can be used to understand and guide appropriate adaptation to a solution in a manner that preserves both the fidelity and real-world generalizability of a solution^66-69 and to guide the pragmatic design, measures, and evaluation of learning cycle findings. By using a TMF and collecting data on relevant contextual issues, learning cycle findings are more likely to be reproduced by others.⁷⁰ For example, RE-AIM^58-61 can be used independently or with another TMF^71-74 for planning and guiding iterative adaptations. An interactive website is available to assist in selecting a TMF that best fits a specific learning cycle or to guide adaptation of a solution.⁷⁵

Pragmatic research solutions.

Efforts are needed to apply pragmatic research methods to QI to maximize relevance and generalizability of learning cycle findings. This has been referred to as improvement science^76,77 in which the contextually relevant issues and methods of QI are augmented by pragmatic research approaches and methods for external relevance. Compared to QI-focused learning cycles, which are often not generalizable, those that leverage pragmatic research approaches and methods will be more generalizable and, therefore, are more likely to be competitive for grant funding and peer-reviewed publication.

Challenge #4: Designing for sustainability

Sustainability of learning cycle solutions and more generally of the LHS infrastructure is imperative to maximize efficiency and impact,⁷⁸ yet it is infrequently identified as a priority. Learning cycles should focus on cost-effective solutions that are feasible to sustain and consider the resources required for maintenance or adaptation over time, including resources needed to de-implement (i.e., unlearn) when value is no longer evident.

Multilevel implementation partner engagement solutions.

To optimize sustainability, LHSs need to consider multilevel contextual factors (e.g., relevant policies, community setting and health system characteristics) and their interactions at local, internal (e.g., heath system characteristics), and external (e.g., clinical practice guidelines, reimbursement models) settings. LHSs should engage partners to identify their priorities and help identify what the LHS’s priorities should be.⁷⁹ As these factors may vary dynamically, the context and relevant partner priorities should be evaluated in depth at the beginning of a learning cycle and iteratively revisited throughout the cycle.^59,80 More broadly, engagement of partners across multiple levels of the socio-ecologic spectrum (patients, the community, clinicians, researchers, leaders, policy makers) is needed to align the LHS with the diverse priorities and practical considerations from each perspective, cultivate a shared understanding of the value of an LHS, and further the sense of investment in the mission and vision of an LHS. However, implementation partner engagement also requires evaluation and transparent reporting of LHS outcomes that are relevant to the different partners’ perspectives; partners should have direct input on what outcomes are meaningful, and these outcomes should be communicated back to them regularly via partner-appropriate dissemination channels.⁸¹

D&I science solutions.

Multilevel context and implementation partner engagement are fundamental components of D&I science, which offers useful TMFs to inform partner engagement activities.⁸² The Stakeholder Engagement Navigator website can assist in deciding what methodologies to use for partner engagement depending on the resources available.⁸³ These methods range from asynchronous online engagement to traditional interviews and more complex consensus-building methods (e.g., modified Delphi). D&I science TMFs can also facilitate sustainability by guiding the evaluation of a solution’s contextual alignment and corresponding implementation strategies.^74,80,84 Other D&I science tools support the concept of designing for dissemination and sustainability, including the PSAT/CSAT – Program/Clinical Sustainability Assessment Tool, which can assist in evaluating institutional sustainability capacity.⁸⁵ Further, D&I science TMFs such as RE-AIM^58,60,61 and implementation outcomes from Proctor et al⁸⁶ can inform locally relevant pragmatic outcomes that consider issues, such as process measures, from multiple perspectives.

Examples of how D&I science can be applied to learning cycles

To assist the reader in understanding how D&I science principles, concepts, and methods can be applied to learning cycles, here we outline 2 real-world examples of D&I science applied to the different steps of learning cycles.

Blood transfusion example.

The objective of this learning cycle was threefold: (1) to understand the extent to which blood transfusions at a single institution are not consistent with evidence-based clinical practice transfusion guidelines, (2) to develop an ordering template to nudge clinicians to order blood transfusion in a manner consistent with clinical practice transfusion guidelines, and (3) to conduct a clinician-level randomized clinical trial of a locally developed CDS tool to test whether the tool’s nudges reduced blood transfusions that are not consistent with guidelines. In the identify gap/problem step, to understand the scope of the problem, we identified the extent to which blood transfusions were not consistent with guideline recommendations and found that up to 50% of blood transfusions were not consistent with guidelines. In the plan/design step, ordering templates that incorporated nudge principles were developed. In addition, implementation partners, including informatics operational leadership and clinicians, were engaged through iterative user-centered design testing to ensure that they had input in the design of the ordering menus. For the do/implement step, a pragmatic, clinician-level randomized trial is in progress, including randomization of all clinicians who order blood transfusions at the institution to receive nudges (intervention condition) or not (control condition). For the study/evaluate step, the Practical, Robust Implementation and Sustainability Model (PRISM)/RE-AIM D&I science framework will be used to pragmatically assess the effectiveness of the ordering menu to reduce blood transfusions that are not consistent with guideline recommendations. As part of the evaluation, a mixed methods evaluation will be conducted to understand the barriers, facilitators, and context of the intervention, which will inform the act/adjust and disseminate findings steps. Key D&I science principles, concepts, and methods that were applied to this learning cycle were implementation partner engagement, selection of a pragmatic study design, and the use of a D&I science framework to define pragmatic evaluation outcomes that facilitate designing for sustainability and dissemination.

Heart failure prescribing example.

The objective of this learning cycle was twofold: (1) to identify approaches to designing a CDS tool that are effective and efficient and (2) to address gaps in guideline-concordant prescribing for patients with heart failure.^87,88 To meet our objectives, we compared a commercially available CDS (i.e., commercial CDS) to an enhanced CDS that was customized for the local health setting context in a cluster-randomized controlled trial within 28 primary care clinics. Commercially available CDSs are generally considered efficient because they require few resources to design, but their effectiveness has been questioned in the literature.^89,90 A specific D&I science framework, PRISM/RE-AIM, was used to identify a pragmatic study design and outcome measures that were aligned with the local context and that would allow for the real-world evaluation of CDS interventions in routine clinical care settings. Both CDSs alerted primary care clinicians during an office visit if they were seeing a patient with heart failure who was not currently prescribed a beta-blocker medication and recommended initiation of this medication. The PRISM/RE-AIM framework was not used to adapt the commercial CDS, but it was used to inform the design of the enhanced CDS so that it was aligned with the local, internal health system, and external (clinical practice guidelines, best practices in designing CDS) contexts. The ways in which D&I science principles, concepts, and methods influenced each step of this learning cycle and improved the enhanced CDS are summarized in Figure 2.

Figure 2.

Example of how D&I science was applied as part of a LHS learning cycle to improve heart failure prescribing. The ways in which D&I science influenced the learning cycle, including the improvement of the enhanced CDS, are called out for each step of this learning cycle. Bolded text in the figure indicates key D&I science principles, concepts, and methods that were applied to this learning cycle. See the “Heart failure prescribing example” section of the text for additional information on this example. Abbreviations: D&I, dissemination and implementation; LHS, learning health system; CDS, clinical decision support; PRISM/RE-AIM, Practical, Robust Implementation and Sustainability Model/Reach, Effectiveness, Adoption, Implementation, Maintenance model.

These 2 examples illustrate different ways and degrees to which D&I science can be applied to learning cycles. In both examples, implementation partner engagement and a D&I science framework were used, but the examples differ in the ways that partners were involved and the extent to which adaptations were proactively considered (i.e., adaptations were not considered in the blood transfusion example). Every learning cycle will apply D&I science differently. Figure 3 summarizes more comprehensively how D&I science could be applied to the various steps of LHS learning cycles.

Figure 3.

Description of how D&I science principles, concepts, and methods can be applied to various steps of LHS learning cycles. Certain learning cycles may not be amenable to certain D&I principles, concepts, or methods. For example, proactive and iterative adaptations may not always be practical or feasible, whereas documentation of unanticipated adaptations is generally possible and key to understanding generalizability and reproducibility of learning cycle findings. Bolded text in the figure indicates key features of this mature LHS model and D&I science principles, concepts, and methods. Abbreviations: D&I, dissemination and implementation; LHS, learning health system; TMF, theory, model, or framework.

Conclusions

Achieving the status of a fully mature LHS is likely to remain aspirational for health systems, but our environmental scan has summarized the challenges to and opportunities for advancing toward this goal with illustrative examples of LHSs at varying stages of maturity (see Table 1). We have identified 4 underrecognized challenges LHSs face: balancing evaluation rapidity with rigor, applying principles of health equity and classic ethics, focusing on external validity and reproducibility (generalizability), and designing for sustainability. To begin with the end in mind as our and other LHSs address these challenges, we have proposed an integrated model of a fully mature LHS (see Figure 1) that includes the key components summarized in Table 2: D&I science, informatics, inclusive approaches to precision health, health equity-focused and multilevel implementation partner engagement, QI, and pragmatic research. This model builds on the work of others, including the National Academy of Medicine, that has sought to identify key characteristics of LHSs.³ The purpose of integrating D&I science with these other well-known methods of informatics, precision health, partner engagement, QI, and pragmatic research is fourfold: (1) to enhance the available data and analytic resources to evaluate care processes (informatics); (2) to ensure that these data include holistic clinical data, including patient-reported outcomes and socio-environmental data (precision health); (3) to address what constitutes success from the perspectives of multiple implementation partners (invested partner engagement); and (4) to increase the speed and dissemination of learning cycle findings (QI and pragmatic research).

Others have emphasized the value of implementation partner engagement, but little emphasis is placed on how to engage invested implementation partners, which is an emphasis of D&I science. Implementation partner engagement should be inclusive and ongoing to account for the dynamic changes that rapidly occur over time in complex health systems.^80,84 Inclusive implementation partner engagement together with inclusive integration of diverse data sources has the potential to maximize health equity outcomes. However, this level of inclusivity is rarely achieved, often because of time or other resource constraints. While inclusive engagement of all partners is important, engagement of partners at higher levels of the socio-ecologic spectrum (e.g., policy) could provide the critical support needed to achieve the full potential of LHSs.

Limitations of this article are that inconsistency in the terminology used by the diverse disciplines involved in LHSs may have biased some of our interpretations of the literature we reviewed. Additionally, we did not conduct a comprehensive systematic review, nor were we able to systematically identify or evaluate LHS work that was unpublished. However, these limitations are mitigated by the triangulation approach we used in our environmental scan that involved a literature review, discussion with LHS experts, and our diverse team representing multiple clinical specialties (cardiology [M.H.], psychology [R.E.G.], primary care [A.G.H., K.E.T.], pharmacy [K.E.T.]); diverse research backgrounds (e.g., D&I science [R.E.G., A.G.H., K.E.T.], pragmatic research [M.H., R.E.G., A.G.H., K.E.T.]); and expertise in QI (M.H., R.E.G., A.G.H., K.E.T.) and informatics (K.E.T.). The scope of this article is also restricted to how D&I science can address the 4 underrecognized challenges for LHSs that we outlined above and does not address the important ways that D&I science can address the well-recognized challenges related to building system-level supporting infrastructure and the accessibility of inclusive, integrated, and actionable data.^6-9 Finally, this article is only an introduction to the value-added contributions of D&I science to LHSs and does not provide in-depth direction on how to leverage its principles, concepts, and methods; however, we have provided examples and extensive references to guide the reader to key TMFs, resources, and papers.

As health care and technology evolve⁹¹ there is a need for ongoing reevaluation of the challenges LHSs face and potential solutions to these challenges. Further, to establish the value of LHSs, there is a need to evaluate and report LHS outcomes, including health equity, reliability, cost, and sustainability outcomes. To facilitate future evaluations, there is also a need to adopt a standard ontology that is understood across transdisciplinary LHS teams. Consistency in terminology can also support transparent dissemination of LHS findings and promote uptake of this aspirational goal for health care delivery.

As an aspirational goal that requires a cultural shift, becoming a mature LHS may seem daunting. Although the target may rapidly change as technology, data availability, and health care advance, a robust and high-value LHS incorporating key approaches from D&I science that consistently delivers equitable patient-centered care is possible. With persistence and collaboration to follow the visionary goals of LHSs, a culture of rapid learning can become the new norm, timely access to actionable and inclusive data will be possible, and significant strides can be made to achieve the Quadruple Aim.