Learning health system benefits: Development and initial validation of a framework

Abstract

Introduction

Implementation of research findings in clinical practice often is not realized or only partially achieved, and if so, with a significant delay. Learning health systems (LHSs) hold promise to overcome this problem by embedding clinical research and evidence‐based best practices into care delivery, enabling innovation and continuous improvement. Implementing an LHS is a complex process that requires participation and resources of a wide range of stakeholders, including healthcare leaders, clinical providers, patients and families, payers, and researchers. Engaging these stakeholders requires communicating clear, tangible value propositions. Existing models identify broad categories of benefits but do not explicate the full range of benefits or ways they can manifest in different organizations.

Methods

To develop such a framework, a working group with representatives from six Clinical and Translational Science Award (CTSA) hubs reviewed existing literature on LHS characteristics, models, and goals; solicited expert input; and applied the framework to their local LHS experiences.

Results

The Framework of LHS Benefits includes six categories of benefits (quality, safety, equity, patient satisfaction, reputation, and value) relevant for a range of stakeholders and defines key concepts within each benefit. Applying the framework to five LHS case examples indicated preliminary face validity across varied LHS approaches and revealed three dimensions in which the framework is relevant: defining goals of individual LHS projects, facilitating collaboration based on shared values, and establishing guiding tenets of an LHS program or mission.

Conclusion

The framework can be used to communicate the value of an LHS to different stakeholders across varied contexts and purposes, and to identify future organizational priorities. Further validation will contribute to the framework's evolution and support its potential to inform the development of tools to evaluate LHS impact.

1. INTRODUCTION

Successful translation from scientific discoveries to implementation in clinical practice and public health takes on average 17 years, ¹ and some research findings are never implemented into care. Deficient or delayed implementation of research findings is detrimental to patient care and health outcomes and contributes to inefficient use of healthcare resources. Barriers to translation in part stem from misalignment of priorities and incentives across stakeholders, including health systems, communities, researchers, and funders. ² The sources of delays extend beyond individual researchers or clinicians to the institutional structures and cultures that have been inadequate for integrating research and clinical care. Ensuring that evidence‐based care practices are implemented necessitates a bidirectional interaction between research and care, informed by active collaboration among health systems, clinicians, patients, and other stakeholders. ³ , ⁴ Learning Health Systems (LHSs) hold promise to strengthen this interplay.

The Institute of Medicine (now the National Academy of Medicine) defines an LHS as “one in which science, informatics, incentives and culture are aligned for continuous improvement and innovation, with best practices seamlessly embedded in the care process, patients and families active participants in all elements, and new knowledge captured as an integral by‐product of the care experience.” ⁵ , ⁶ LHSs bridge the generation of evidence and its implementation by integrating local findings with existing knowledge and applying these cumulative findings with expediency and flexibility as more information is gathered. ⁷ Importantly, the structure and operational approach of LHSs vary widely. ⁸ Much focus has been on implementation approaches, theory, and policy, with relatively few articles discussing the importance of the social and ethical aspects to stakeholders. ⁹

The LHS concept is evolving within a US healthcare system already using continuous quality improvement (CQI) methods and dissemination and implementation (D&I) science and practice to improve health care and outcomes. CQI programs seek to achieve quality goals, improve operations, maintain accreditations, and meet quality metrics and reporting requirements. D&I science and practice seek to facilitate the application of evidence‐based knowledge in real‐world, routine care. ¹⁰ As CQI and D&I often have similar aims and strategies, greater integration could enhance the speed, effectiveness, and impact of organizational learning. ¹¹ Ideally, an LHS facilitates synergies across CQI efforts, D&I science and practice, and research capabilities, such as those supported by the Clinical and Translational Science Award (CTSA) program. This integration optimizes data‐driven learning for iterative improvement. ¹²

Developing and growing an LHS is a complex process involving science; human behavior; and the participation, resources, and investments of many stakeholders over time. Executives, administrators, frontline clinicians, patients and families, payers, and researchers are all indispensable to the success of an LHS. Furthermore, LHSs require strategic planning and preparatory work to bolster buy‐in from these stakeholders ¹³ and define clear goals and outcomes that will build institutional readiness and the culture shift that enables success. Ensuring each stakeholder group is represented, visible, and aligned (early and often) is paramount. To achieve a collective sense of commitment, multidirectional dialogue and effective engagement strategies need to be appropriately tailored toward building a shared vision and understanding, as stakeholders often will bring disparate but valuable perspectives, needs, and wants. ¹⁴

The CTSA Program is uniquely positioned to ameliorate LHS roadblocks, including comprehensive stakeholder engagement. ² In 2021, representatives from six CTSA hubs convened a working group to share experiences with supporting their local LHSs. Although these hubs' local healthcare organizations had varied approaches to and levels of maturity of LHSs, they had a shared need to clearly articulate specific and tangible value propositions to a range of stakeholders, ¹⁵ whose interests and needs can evolve over time.

1.1. Question of interest

To begin addressing this shared need across different LHS models, the working group asked: What is the full range of anticipated and/or realized benefits of an LHS that can be used to create effective value propositions that will attain and sustain commitment and support, including resources, from diverse stakeholders? To our knowledge, no existing LHS model provides a structured, operationalized framework of potential benefits and ways those benefits can manifest in different organizations. This article proposes a framework to fill this gap.

2. METHODS

Framework development was primarily based on literature review of existing LHS models, expert input, and the interface of five of the working group CTSA hubs and their local LHSs.

Leveraging existing scholarship, the working group began by examining a 2022 systematic literature review intended to specify the types of work and enabling conditions associated with LHSs for healthcare delivery organizations. ¹⁶ This existing review included 79 LHS publications, served as the basis for developing the Learning Health Systems Consolidated Framework (LHS‐CF), and noted that four LHS goals emerged from the review (quality, value, safety, equity).

The working group also conducted a supplemental limited literature search of academic health sciences publications and gray literature to identify models and articles that explicated LHS benefits. Specific sources that were reviewed included the Kaiser Permanente Washington (KPWA) LHS Logic Model, ¹⁷ the Framework for Value‐Creating Learning Health Systems, ¹⁸ the Learning Health System Framework and accompanying report, ¹⁵ , ¹⁹ the Quintuple Aim for Healthcare Improvement, ²⁰ , ²¹ and a study detailing patient perspectives on LHS projects. ²² Additional models that were reviewed focused on the components of LHSs but did not speak directly to specific LHS benefits. ²³ , ²⁴ , ²⁵ , ²⁶ , ²⁷

Given the sources used varying language to refer to similar concepts, the working group used a consensus process to identify terms for general categories and key concepts within them, ensuring the perspectives gained from the sources reviewed were represented. The working group drew on published literature to define the key concepts. Finally, an expert external to the working group reviewed and commented on the framework's categories, key concepts, and definitions, and the working group revised based on feedback.

An initial step toward validating the framework entailed assessing its usability and face validity by determining the extent to which five LHS “cases” represented in the working group reflected the framework's benefits. Each case included four standard elements: types of LHS work conducted, genesis and motivations to pursue an LHS model, current status of the LHS evolution, and any ways in which the case reflected the framework's benefits.

3. RESULTS

The review of existing models and other literature revealed that many models depicted the components of an LHS; few included LHS benefits; and none of the reviewed models consolidated the full range of benefit categories, specified key concepts within categories, and provided definitions. Building on existing literature, the working group created the Framework of Learning Health System Benefits (Figure 1) to support effective communication of value propositions to a broad range of stakeholders. The framework includes six categories of benefits: quality, ¹⁶ , ¹⁷ , ¹⁹ equity, ¹⁶ , ¹⁹ safety, ¹⁶ , ¹⁹ , ²² patient satisfaction, ¹⁷ , ¹⁹ organizational reputation, ¹⁹ and value. ¹⁶ , ¹⁹ For each category, the framework delineates key concepts within that benefit. The different LHS stakeholders are centered in the framework to indicate that each type of benefit is relevant to all groups. Importantly, each stakeholder group may prioritize different benefits, and it is expected that benefits most relevant to, or prioritized by, an LHS may shift over time.

FIGURE 1.

Framework of Learning Health System Benefits.

Although not exhaustive, the key concepts provide different ways in which each benefit may manifest within an organization or be experienced by stakeholders (Table 1). The broad category of quality includes both optimized care that implements and de‐implements processes and treatments as appropriate and the achievement of desired outcomes. Equity manifests through fair, just, and impartial access to quality care and through inclusiveness in clinical research. Safety entails fostering a culture of safety, proactive risk reduction, and achievement on known safety risk areas. Value can be achieved by cost‐effective care (outcome relative to cost), reduction in cost with no change in quality, and improvement in societal and economic costs of illness. Satisfaction manifests as a patient's subjective experience of healthcare interactions or a more objective assessment of whether care is patient‐centered. Finally, reputation is an important category for healthcare organizations that is reflected in its ability to attract and retain patients, clinicians, and staff, and the degree to which employees are engaged and invested in their work.

TABLE 1.

Definitions of concepts by benefit category.

Concept	Definition
Quality
Optimized care	Healthcare that routinely implements effective (ideally evidence‐based) and reliable processes and treatments, and de‐implements ineffective ones 43
Desired outcomes	Positive change in health indicators and/or health‐related quality of life resulting from healthcare provided, including but not limited to publicly‐available metrics of quality (eg, readmission rates, mortality, etc.)
Equity
Broad access to care	The fair and just availability of, and impartial ability to obtain, quality healthcare across all identity groups (eg, gender, sex, sexual orientation, race, ethnicity, ability), age groups, socioeconomic groups, and geographic locations, 44 including NIH‐designated US health disparity populations. 45
Inclusive clinical research	Development and use of an evidence base related to health and healthcare that represents all identity groups, age groups, socioeconomic groups, and geographic locations, including similarities and differences at multiple levels (population, community, and individual provider or patient) 46 , 47 , 48
Safety
Culture of safety	Shared values, beliefs, and attitudes that prioritize and reward the prevention of harm resulting from healthcare; includes shared responsibility for mistakes and continuous learning 49 , 50
Risk reduction	Proactive implementation of processes and practices that reduce the chances of exposing patients, clinicians, and other staff to events that could have resulted in physical or psychological injury. 51
Safety performance	Achievement on known safety risk areas through safety compliance and safety participation; typically measured by key indicators that may also be used to benchmark progress across organizations 52
Value
Cost effectiveness	Comparison of the desired outcomes of health care for a condition relative to the financial outlay for the full cycle of care for the condition 53 , 54 , 55 , 56 , 57
Cost savings	Reduction in the financial outlay to obtain or provide health care with no effect on quality (eg, use of generic medication) 58 , 59
Lower and economic costs of illness	Improvement in social and financial indicators as a result of health care, including quality of life and worker productivity 6 , 60
Satisfaction
Patient experience	The collection of interactions that patients have with providers, staff, and facilities in a healthcare system; includes assessing whether standards that patients value are met (eg, clear and responsive communication with providers, ability to schedule timely appointments, straightforward information sharing) 61
Patient‐centered care	Health care that is compassionate, empathetic, and responsive to the needs, values, and expressed preferences of each individual patient; includes patients as shared decision makers in their care 50
Reputation
Competitive within market	Ability of a healthcare organization to attract patients, clinicians, or other staff due to a general perception that it has an equal or greater level of desired attributes (such as quality care, ethical conduct, social responsibility, research) than similar organizations in its region 62
Employee engagement	The degree to which clinicians and staff have a positive attitude toward, and invest their talents and energies in, their work, their teams, and the organization's goals. 63 The care team experience in the organization is a driver of attitude and engagement.

The following five case studies demonstrate a variety of organizations' approaches to building an LHS and how consideration of benefits played out in those approaches.

3.1. Case 1: Initiating a Patient‐centered Learning Health System

Tufts Medicine is in the early stages of developing its Patient‐centered Learning Health System (PCLHS). As an integrated academic health system, Tufts Medicine brings together an outpatient physician network, home health and hospice care, two community hospitals, and an academic medical center. Together with Tufts Clinical and Translational Science Institute (CTSI), Tufts Medicine is seeking to more effectively use internal data across these varied clinical settings to generate evidence that supports continuous learning and improvement. Each hospital within the system is engaged in research and/or quality improvement, but previous studies had not necessarily focused on systematically gathering and applying evidence in real time to guide care. The PCLHS initiative is using the research and quality improvement building blocks already in place to establish synergies that address clinical and operational goals. The initiative uses the term “patient‐centered” to emphasize an essential LHS element: the core focus is on the patient and learning that benefits their health.

Initial LHS activities include the establishment of a steering committee and the initiation of two foundational projects. The steering committee, which includes leadership in the areas of informatics, quality and safety, clinical care, research, and system integration, is in the process of determining an approach to LHS activities that fits the system's organizational structure and operational goals. Next steps in operationalizing the PCLHS will be to formalize the infrastructure supporting it and to determine how future initiatives will be selected, executed, and sustained.

An important part of operationalizing the PCLHS will be to more clearly articulate the specific benefits stakeholders aim to achieve. To date, Tufts Medicine's advancement as an LHS has been driven by the benefits its patients will realize through advances in quality, safety, and equity. For example, one of the initial LHS projects uses a newly developed artificial intelligence (AI) and machine learning platform to develop an effective predictive model to identify patients with heart failure at risk for readmission, who will then receive an enhanced readmission prevention protocol. A second project is establishing a model for N‐of‐1 clinical trials that might be integrated into general care to determine the most effective treatment for individual patients. By identifying and addressing patient‐specific characteristics and healthcare needs, PCLHS projects aim not only to support clinical quality and safety but also to mitigate persistent healthcare biases and enhance health equity.

3.2. Case 2: Leveraging an existing CTSA research infrastructure for clinical intervention

In 2018, South Carolina rural communities increased their healthcare access with the expansion of the Medical University of South Carolina (MUSC) Regional Health Network (RHN). With the onset of the global COVID‐19 (COVID) pandemic, the local urgency to offer palliative treatments during the pre‐vaccine months required a seismic shift in the traditional implementation processes of the existing infrastructure. The South Carolina Clinical and Translational Research (SCTR) Institute, as part of the CTSA Program, had primarily focused on innovation and quality impacts within the research translation continuum. The immediacy of COVID compressed the strategic integration timelines between MUSC RHN, MUSC Research Administration, SCTR Institute, and key stakeholders. The accelerated integration of research infrastructure and a rural clinical network developed cutting edge, innovative processes to bolster equity and quality to COVID care. The MUSC Regional Health Network's immediate response to the health crisis also bolstered its organizational reputation as an invested and trustworthy community partner.

Rural communities were one of the hardest hit by rising COVID infection rates with diminished care options due to the closure of extension hospitals. There was also mistrust in the traditional medical establishment, ²⁸ , ²⁹ , ³⁰ where many were reluctant to seek clinical care outside of their local areas, ³¹ and often had minimal understanding of and participation in clinical trials. Leveraging SCTR and MUSC management models, the existing research support infrastructure was able to assist the MUSC RHN in becoming research‐ready and compliant in order to offer COVID‐related clinical trial interventions.

Activities coordinated through SCTR‐funded programs included developing a centralized, web‐based resource for up‐to‐date research and clinical policies, and providing remote consenting and screening standard operating procedures, safety workflows and documentation, and access to trainings for good clinical practice and other research‐related compliance requirements. SCTR's Remote Clinical Trials Design service enabled virtual trials and provided guidance in designing remote trial protocols, and the existing biorepository began to collect, process, and house longitudinal blood and saliva samples.

As a result, RHN clinicians were able to offer patients COVID trial interventions and local hospital study staff efficiently complied with research requirements. The new “buy‐in” and goodwill generated other possibilities for future trial opportunities. As the MUSC RHN grows, MUSC Research Administration and SCTR intend to scale these research integration processes to meet emergent locally identified needs.

3.3. Case 3: Building on a history of clinical continuous quality improvement

The University of Rochester (UR) Medicine's LHS work focuses on building research into CQI projects, including, as appropriate, effectiveness and/or dissemination and implementation components. In 2019, parallel initiatives in the clinical and research domains united to support the LHS's development and continual evolution.

Rochester's CTSI had a longstanding interest in supporting an LHS (termed Meliora [“Ever Better”] Health) as part of the translational research continuum to advance the adoption of evidence‐based practices across the clinical enterprise. ¹² Specifically the Equity focused‐Dissemination and Implementation (EQ‐DI) core sought to spearhead LHS development and progress through effective translation, distribution, and use of evidence‐based interventions and policies in real‐world settings, with an explicit focus on addressing health inequities.

Relatedly, the clinical enterprise had a decades‐long commitment to data‐based decision‐making and improving quality. However, the initiatives were disparate, and some were methodologically weak (ie, unclear outcomes, inadequate consideration of implementation challenges, evaluation lacking), leading to unclear impacts, inability to ascribe changes to the intervention, and lack of readiness for scale‐up. Furthermore, the clinical data system was not sufficiently nimble, and some data quality concerns remained (eg, lack of social determinants of health data).

In 2019, the Quality Institute was established to harness, coordinate and prioritize ongoing CQI work based on guiding principles of health equity, patient and family centered care, and clinical ethics. The Quality Institute's Research and Evaluation pillar formally incorporated the LHS work. Including both research and clinical faculty, the pillar establishes systematic processes for evaluation design, project identification, and prioritization, and it supports internal and external dissemination and milestones to assess progress. The initial challenge was adequate engagement with and buy‐in from the clinical enterprise.

Despite pandemic‐related delays, progress on integration has included consensus on an LHS definition and acknowledgment of the aforementioned limitations, challenges, and complexities of dissemination and implementation. CQI work that had focused previously on hospital processes is expanding to include ambulatory projects. On the research side, the CTSI recognizes the sensitivity in the clinical enterprise about doing “research,” often viewed as adding burden but not value. A re‐engineered data infrastructure now provides easier access to clinical data to inform quality initiatives. Social determinants of health assessments are now built into the system with dashboards to identify disparities and opportunities for CQI work.

The Quality Institute seeks to maximize value, satisfaction, reputation, equity, and quality. These mirror the LHS benefits framework and, along with shared values, provide a common foundation to effectively and systematically integrate Meliora Health into clinical CQI.

3.4. Case 4: Operationalizing an LHS research support platform for pragmatic trials

As part of Vanderbilt University Medical Center's journey toward realizing an LHS, the local CTSA, Vanderbilt Institute for Clinical and Translational Research (VICTR), strategized on infrastructure that could empower research efforts within this space. Capitalizing on established critical care pragmatic trials excellence ³² , ³³ , ³⁴ , ³⁵ , ³⁶ and a desire to expand that methodologic approach, VICTR crafted a blueprint for an LHS‐supportive programmatic offering to bring research and operations together as collaborative partners. Preliminary discussions were bolstered by involvement and investment of key champions, including the Executive Vice President for Clinical Research/VICTR Director, the Executive Vice President for Public Health and Health Care, and the hospital's Chief Executive Officer and Chief Nursing Officer.

Consensus emerged with emphasis on weaving cycles of discovery and improvement into the fabric of routine care delivery and hospital operations. At the outset, the LHS Platform targeted opportunities to optimize outcomes (quality), increase patient satisfaction, and maximize value while providing the most effective, efficient, and safe care. ³⁷ , ³⁸ The resulting prospective, pragmatic randomized controlled trial (pRCT) research support platform coalesced in 2017 and continues to target comparative effectiveness examinations to drive generation of evidence‐based best practice recommendations.

The LHS Platform pRCT engine ensures technical, procedural, and human infrastructure is in place to promote, develop, and support LHS‐aligned ventures—shepherding ideas through a programmatic pipeline supporting various levels of need from initial study conception through to study close‐out. This framework requires a transdisciplinary, team science approach with strengths in clinical trial design and conduct, biostatistics, regulatory affairs, project management, data science and informatics, healthcare administration and operations, and community engagement. The Platform convenes monthly steering and quarterly strategic committee meetings to evaluate project concepts, review progress, assess programmatic needs, analyze barriers and facilitators, and establish priorities. ¹⁴

At the close of its first 5 years, the LHS Platform supported and/or assisted 103 projects: 32 pragmatic clinical trials were ongoing, 13 published/completed, 26 received substantial design input but were placed on‐hold for various reasons, and 32 received consultation but were not amenable to a pragmatic RCT design. The Platform is now prioritizing the methods required to transform findings into sustainable practice change in partnership with patients, providers, and the health system. Additionally, though studies supported by the LHS Platform approach design thoughtfully to ensure patient population representation and inclusivity, intentional design to reduce inequity is imperative. Questions of interest to a broader group of stakeholders and that are purposeful in their equity goals are being pursued.

3.5. Case 5: Growing an inter‐organizational statewide LHS collaboration

Recognizing the need for inter‐organizational and multidisciplinary collaboration to improve healthcare across Indiana, the Regenstrief Institute set the goal of establishing the Indiana Learning Health System Initiative (ILHSI) ³⁹ as a statewide (regional) LHS involving multiple health systems in 2017. Its foundational vision was that high‐quality care should be available to anyone in Indiana, regardless of location, health system, or individual circumstances.

An executive steering committee (ESC) with representation from six organizations (the Regenstrief Institute, Indiana University [IU] Health, Eskenazi Health [EH], the IU School of Medicine [IUSM], the Fairbanks School of Public Health, and the Indiana Health Information Exchange) was charged with developing a plan. The ESC determined that LHS pilot projects would be driven by: (i) business unit or clinical need emerging from health system committees or units with which ESC members engaged, and (ii) operational or clinical projects considered to be of strategic importance to the executive leadership of the partner health systems.

The ESC also engaged approximately 40 additional stakeholders to discuss key structures, foci, and goals. These stakeholders included various health systems' executive, clinical, and operational leadership; research leaders at the IUSM; and healthcare‐ and community‐focused researchers, informatics researchers, and data analytics and business development representatives at the Regenstrief Institute. A Program Team (PT) functioned as the central, operational component for planning, managing operations, and identifying projects and funding sources, from the initiation of the ILHSI and going forward.

Initial ILHSI projects addressed the benefits of quality and value, as well as, partially, safety. Most have focused on improving care processes or outcomes by using a strong informatics approach and also have attempted to extend our knowledgebase where possible. As one example, the Health Dart project integrated selected, high‐value data from Indiana's health information exchange relevant to a patient's chief complaint directly with IU Health's electronic health record, and has been rolled out across all 15 Emergency Departments in the IU Health system. Another project developed a predictive model of early in‐hospital mortality risk and is currently evaluating its implementation for guiding critical illness conversations and decisions.

Building on the lessons of the first 4 years, ILHSI's next phase is characterized by (1) seed funding for the LHS partnership supplied by IU Health to support infrastructure and preliminary studies to bolster external funding applications; (2) a streamlined administrative, project management, and oversight structure; and (3) identifying and implementing cross‐health system projects focusing on a shared interest in equity.

4. DISCUSSION

A cross‐CTSA working group identified a need for a framework to assist in formulating value propositions for various LHS stakeholders that was not fully met by the representation of LHS benefits in existing LHS models. To fill this gap, the group built on insights from existing models to develop a structured Framework of Learning Health System Benefits. The framework consolidates the range of currently recognized LHS benefits, further specifies and defines common ways each benefit may be manifested, and centers the variety of stakeholder groups to consider when crafting value propositions. As a first step toward assessing the framework's relevance and face validity, the working group applied it to five varied case studies of LHS approaches. This assessment demonstrated three dimensions of LHS work in which the framework is relevant and appears to have face validity: defining goals of individual LHS projects (Tufts Medicine, Medical University of South Carolina), facilitating collaboration based on shared values (University of Rochester), and establishing guiding tenets of an LHS program or mission (Vanderbilt University Medical Center, Indiana Learning Health System Initiative).

Although the case studies demonstrated the relevance of the framework, the LHSs represented were initiated prior to the development of the framework of benefits. As a result, the cases do not fully capture its prospective utility to guide decision making or gain buy‐in from stakeholders. The variety of benefits represented in the tool speak to the value proposition for a range of stakeholders. Intentionally discussing desired benefits with stakeholder groups may bring to light competing priorities that could hamper LHS progress if not addressed. This applies to individual LHS projects as well as overarching missions. Some benefits may be deemed of higher priority, depending on an LHS' strategic goals or a desire to demonstrate “early wins.”

The framework also can be used to plan for future goals or priorities for the growing or maturing LHS. Notably, none of the case studies addressed all of the framework's benefits at a single point in time. Planning for areas of expansion or growth supports continued interest and investment in an LHS, and may engage a wider array of stakeholders. Desired benefits that fueled the initial LHS development may change as stakeholders' focus shifts to expanding or sustaining an LHS model.

4.1. Limitations and future development

While the case studies preliminarily demonstrated the relevance and face validity of the framework, more formal application and testing is needed. This testing should seek to confirm face validity in other settings, validate the content, substantiate applicability of the tool across contexts and LHS models, and confirm or extend the various prospective uses suggested here. An iterative process of refining the tool over time is common for model development ²⁰ , ²¹ , ⁴⁰ , ⁴¹ and can account for changes in practical experience, such as changes in approaches in LHS development and maturity.

We anticipate that a useful extension of the framework of benefits will be to inform the development of a tool to evaluate an LHS' impact. Measuring and effectively demonstrating the extent to which intended benefits are achieved in an organization, and the stakeholders experiencing those benefits, may be valuable for sustaining investment in LHS activities. Although a full consideration of impact is beyond the scope of this manuscript, we recognize a potential connection with the existing impact assessment tools. For example, the Translational Science Benefits Model (TSBM) ⁴² provides a systematic, broad‐based approach to compiling and documenting impact of clinical and translational research. Given its focus on research studies, though, not all TSBM indicators may be relevant to LHS impact, and there may be LHS benefits (eg, organizational reputation, patient satisfaction) that go beyond the TSBM indicators. Future development of an LHS impact evaluation tool informed by the intersection of the framework of benefits and the TSBM could provide a useful approach to documenting and communicating the broad impact potential of an LHS.

FUNDING INFORMATION

This project was supported by the National Center for Advancing Translational Sciences, National Institutes of Health, Award Numbers: UL1TR002529 and UM1TR004402, The Indiana Clinical and Translational Sciences Institute; UL1TR001450, South Carolina Clinical and Translational Research Institute, Medical University of South Carolina; UL1TR002544 and UM1TR004398, Tufts Clinical and Translational Science Institute, Tufts University; UL1TR002001, The University of Rochester Clinical & Translational Science Institute; UL1TR002243, Vanderbilt Institute for Clinical and Translational Research; UL1TR001420, Wake Forest University School of Medicine Clinical and Translational Science Institute. The Lilly Endowment, Inc., Physician Scientist Initiative, and the Advances in Medicine (AIM) grant from Cook Medical provided additional support. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or other funders.

CONFLICT OF INTEREST STATEMENT

The authors declare no conflicts of interest.

Welch LC, Brewer SK, Schleyer T, et al. Learning health system benefits: Development and initial validation of a framework. Learn Health Sys. 2024;8(1):e10380. doi: 10.1002/lrh2.10380