The Indiana Learning Health System Initiative: Early experience developing a collaborative, regional learning health system

Abstract

Introduction

Learning health systems (LHSs) are usually created and maintained by single institutions or healthcare systems. The Indiana Learning Health System Initiative (ILHSI) is a new multi‐institutional, collaborative regional LHS initiative led by the Regenstrief Institute (RI) and developed in partnership with five additional organizations: two Indiana‐based health systems, two schools at Indiana University, and our state‐wide health information exchange. We report our experiences and lessons learned during the initial 2‐year phase of developing and implementing the ILHSI.

Methods

The initial goals of the ILHSI were to instantiate the concept, establish partnerships, and perform LHS pilot projects to inform expansion. We established shared governance and technical capabilities, conducted a literature review‐based and regional environmental scan, and convened key stakeholders to iteratively identify focus areas, and select and implement six initial joint projects.

Results

The ILHSI successfully collaborated with its partner organizations to establish a foundational governance structure, set goals and strategies, and prioritize projects and training activities. We developed and deployed strategies to effectively use health system and regional HIE infrastructure and minimize information silos, a frequent challenge for multi‐organizational LHSs. Successful projects were diverse and included deploying a Fast Healthcare Interoperability Standards (FHIR)‐based tool across emergency departments state‐wide, analyzing free‐text elements of cross‐hospital surveys, and developing models to provide clinical decision support based on clinical and social determinants of health. We also experienced organizational challenges, including changes in key leadership personnel and varying levels of engagement with health system partners, which impacted initial ILHSI efforts and structures. Reflecting on these early experiences, we identified lessons learned and next steps.

Conclusions

Multi‐organizational LHSs can be challenging to develop but present the opportunity to leverage learning across multiple organizations and systems to benefit the general population. Attention to governance decisions, shared goal setting and monitoring, and careful selection of projects are important for early success.

1. INTRODUCTION

The implementation of learning health systems (LHSs) is an area of strong interest for healthcare organizations, policymakers, and the public. ¹ , ² , ³ To date, however, many LHS efforts have been developed within single organizations, ³ , ⁴ , ⁵ , ⁶ , ⁷ , ⁸ , ⁹ , ¹⁰ , ¹¹ , ¹² , ¹³ not as collaborations among separate entities. Given the need to develop collaborative LHSs that are healthcare market‐centric, as opposed to organization‐centric, two questions are particularly relevant: (a) What are promising approaches to operationalizing an LHS across a regional healthcare market that spans different organizations? (b) How can multiple organizations, such as healthcare systems, research institutes, and others, collaborate to create an LHS ecosystem?

LHS initiatives can be, roughly, grouped into three categories with regard to their scope: (a) single organization, (b) trans‐organizational, and (c) multi‐organizational/healthcare market‐focused:

1. Single‐organization LHSs commonly make the LHS approach a key strategic and operational philosophy driven by top‐down priorities. Examples include the national health systems of Switzerland ¹⁴ and Denmark, ¹⁵ the Veterans Health Administration (VHA), ³ , ⁴ , ⁵ , ¹³ or multi‐site and single healthcare organizations such as Kaiser Permanente, ⁶ , ⁷ the Geisinger Health System, ⁸ and others. ⁹ , ¹⁰ , ¹¹ , ¹² While a common organizational foundation can promote the development of LHS approaches, significant cultural, and scalability challenges can impede their dissemination throughout all levels and sites. ¹⁶ , ¹⁷ , ¹⁸

2. Trans‐organizational LHSs include more than one organization and are typically focused on a shared, focused purpose. Examples include programs that evolved from existing registries for specialty areas, such as the CERTAIN LHS in Washington state, ¹⁹ and several disease‐specific initiatives. ²⁰ , ²¹ While these initiatives have greater opportunities for addressing health needs than single organizations, they are often challenged by data interoperability, privacy, market challenges, and inter‐organizational dissemination. ¹⁷

3. Multi‐organizational/healthcare market‐focused LHS initiatives are collaborative endeavors that span multiple distinct healthcare organizations across a region/healthcare market. They leverage a shared infrastructure, processes, and collaboration for common outcomes. The only published LHS effort in this category is Health Sciences South Carolina (HSSC), founded in 2004. ²² HSSC “is the nation's first statewide health collaborative and is committed to transforming South Carolina's public health and economic well‐being through research.” A significant focus of the HSSC has been the development of a common data infrastructure. In our case, this data infrastructure, the Indiana Network for Patient Care (INPC), including governance, policies, and operations, already existed. ²³

The Indiana Learning Health System Initiative (ILHSI) is a new multi‐organizational and healthcare market‐focused LHS initiative. Initiated and led by the Regenstrief Institute (RI), the ILHSI is being developed in partnership with five additional organizations: two health systems, Indiana University Health (IUH) and Eskenazi Health (EH); two schools of Indiana University, IU School of Medicine (IUSM) and IU Fairbanks School of Public Health (FSPH); and the Indiana Health Information Exchange (IHIE) (Table 1). Building on statewide infrastructures and partnerships among academic, clinical, and data systems, ²³ the ILHSI's long‐term vision is to include multiple health systems across Indiana (Figure 1). As such, the design of the ILHSI adheres to the “think globally, act locally” principle for creating local LHSs we previously described, ¹⁸ and follows the philosophy “that achievement of a national‐scale LHS will not be the work of a single organization, stakeholder group, or governmental entity. Rather, it is anticipated that the LHS will require active participation of and cooperation among multiple and diverse stakeholders, nationwide and ultimately globally”. ²⁴

TABLE 1.

The six foundational partners of the Indiana Learning Health System Initiative include a research institute (Regenstrief Institute); two health systems, Indiana University Health and Eskenazi Health; two schools of Indiana University, IU School of Medicine and IU Fairbanks School of Public Health; and the Indiana Health Information Exchange

Organizational partner	Role
Regenstrief Institute (RI)	The RI is a nonprofit research institute affiliated with the IUSM with deep experience in informatics, health services research, aging research, and implementation science that functions as the central driver and coordinator of ILHSI development. It serves a critical function in identifying and bridging operational silos within IUH and other healthcare organizations that are often barriers to extending LHS activities.
Indiana University Health (IUH)	IUH is an academically‐based, statewide, integrated health system comprised of an Academic Health Center, critical access hospitals, and community hospitals and clinics sharing clinical, operational, administrative, and information technology infrastructure and practices. This structure provides the opportunity for a statewide reach for the ILHSI and a living laboratory for LHS initiatives in diverse settings and populations.
Eskenazi Health (EH)	Eskenazi Health is one of America's five largest safety net health systems. It is a tax‐supported, urban healthcare system that provides outpatient, inpatient, and community‐based health services to residents of Marion County, Indiana. Eskenazi Health has a long‐standing research partnership with RI.
Indiana University School of Medicine (IUSM)	The IUSM is the only allopathic medical school in Indiana, with regional campuses anchored by the main campus in Indianapolis. The IUSM supports the development and integration of academic medicine and research into community healthcare systems.
Fairbanks School of Public Health (FSPH)	The FSPH provides academic programs that focus on public health and healthcare administration, and include undergraduate and graduate degrees. The research efforts of FSPH focus on improving the health of communities. In the Center for Health Policy, faculty and staff collaborate with state and local government, as well as public and private healthcare organizations, to conduct high‐quality program evaluation and applied research on critical health policy‐related issues.
Indiana Health Information Exchange (IHIE)	The IHIE develops and maintains the Indiana Network for Patient Care (INPC), one of the nation's largest interorganizational clinical data repositories. The INPC addresses critical operational challenges inherent in a trans‐organizational LHS initiative, as it aggregates data from disparate healthcare systems throughout Indiana, serving both operational (direct patient care) and research needs. Well‐established, long‐standing data governance agreements among IHIE members facilitate data analysis and reuse in support of LHS activities.

FIGURE 1.

The Indiana Learning Health System Initiative, initiated and led by the Regenstrief Institute, is a collaborative of patient care, research and academic, and health information exchange and data partners. It builds on multiple research () and research‐and‐data relationships ()

The primary objectives of this initial phase of the ILHSI were to instantiate the concept, establish collaborative partnerships, and pilot LHS activities in support of developing and implementing a regional, and eventually state‐wide, trans‐organizational LHS. Our approach was based on the lessons learned by other large health systems and organizations that have implemented an LHS approach, as well as existing frameworks. ³ , ²⁵ Specifically, the VHA Quality Enhancement Research Initiative (QUERI) programs framework, outlined in 2019, was developed to accelerate the adoption and spread of evidence‐based practices across the VHA, and is based on the previous more linear “pipeline” framework for implementation. ²⁶ As such, it conceptualizes the dynamic process of learning into a pragmatic LHS framework that is adaptable across organizations with varied structures and cultures, and assists the process of implementation by broadly organizing key questions, strategies, and measures into Pre‐implementation, Implementation, and Sustainment phases. Our development of the IHLSI was based on this general framework, with core activities and structures linked to these phases of implementation. To our knowledge, this paper is only the second reported analysis of a process to establish a multi‐organizational and healthcare market‐focused LHS.

The ILHSI is a work in progress, and has not yet resulted in a fully developed and implemented multi‐organizational LHS. Rather, we present our early findings and reflections on the pre‐implementation and initial implementation phases of work with selected health systems to help others planning LHS efforts that transcend organizational boundaries, as well as contribute to the literature on the development of LHSs. Our primary questions of interest were: What was learned in the early phases of establishing a multi‐organizational and healthcare market‐focused LHS? What specific lessons were learned about building a productive, sustainable collaboration? What are challenges to and next steps in LHS maturation across a region?

2. METHODS

2.1. Setting

Recognizing the need for interorganizational collaboration to combine informatics, clinical operations, and research to improve healthcare across Indiana, RI established the ILHSI as a new strategic initiative in 2017. This goal was driven by the institute's recently appointed president and CEO (coauthor P.E.), who was appointed as vice‐president for LHSs at IUH, a statewide academic health center that partners with IUSM, in addition to his other roles at IUSM, and with the Indiana Clinical and Translational Science Institute. Dr Embi's roles recognized the need to align the LHS concept across partner organizations.

2.2. Pre‐implementation activities

We first established and charged a Planning Committee (PC), and appointed members from across the six organizations (RI, IUH, EH, IUSM, FSPH, and IHIE) (Table 1) who had both the expertise and authority to advance the ILHSI. Members included: two RI‐based champions with informatics and implementation science expertise (coauthors T.S. and L.W.) to help lead informatics and implementation science aspects of the initiative; the Chief Medical Officer (then coauthor J.G.), Senior Vice President of Clinical Effectiveness (coauthor C.W.), Vice President Quality, Safety, and Performance Improvement (coauthor M.S.), Chief Information Officer, and Chief Medical Information Officer (CMIO) of IUH; the CMIO, and Chief Research and Development Officer, of EH; the CEO of IHIE; and three additional scientists from RI who are also faculty at IUSM or FSPH with expertise in health informatics, health services research, aging research, and implementation science. This group started by conducting pre‐implementation interviews focusing on identifying core assets, existing collaborations, and shared goals. A literature review followed to identify examples of existing LHS efforts, particularly those focused on interorganizational efforts, and studies of successful structures and frameworks. ⁵ , ²⁵ We then engaged in an iterative process of collaboratively determining the initial phase of work, including some early projects that would be identified from two complementary directions. Recognizing the benefits and limitations of both top‐down and bottom‐up approaches, we worked to identify LHS pilot projects that would be driven by (a) business unit or clinical needs emerging from health system committees or units with which our PC members are engaged, and (b) operational or clinical projects considered to be of strategic importance to the executive leadership. Critical questions addressed during this process included the following: What steps should RI take to work with its health system partner in implementing a multi‐year LHS initiative? What existing projects/initiatives, either at RI or participating health systems, fit into an LHS strategy? How could the institute help support those? And what near‐term, joint projects could be considered for launching the initiative?

In addition, the PC engaged approximately 40 additional stakeholders to discuss key structures, foci, and goals for ILHSI development. These stakeholders included executive, clinical, and operational leadership at IUH, such as leaders in quality, patient safety, and clinical analytics; representatives from other health systems, such as the US Department of Veterans Affairs VA); research leaders at the IUSM; and healthcare‐ and community‐focused researchers, informatics researchers, and representatives from data analytics and business development at RI. Selection of these individuals was based on consensus of the PC, with the goal of involving potential partners and other interested parties in the design stage, to incorporate their interests and concerns, identify ongoing LHS‐related activities and programs, and promote future buy‐in. These stakeholders provided input through individual meetings and interviews, group meetings, sharing of documentation about existing local LHS efforts, and feedback on document drafts.

Critical questions addressed during this process included the following: What are the key foci critical to initial development and future sustainability and spread of the ILHSI? How can the ILHSI enhance existing culture, operations, and clinical care in healthcare systems and the marketplace? What challenges could impede efforts to establish and spread LHS activities throughout the community, and how could the ILHSI help address these challenges? Notes from our PC and stakeholder group interviews were summarized and analyzed by the ILHSI leadership (P.E., T.S., L.W.) to abstract findings and develop draft plans that were then reviewed and approved via a consensus process by the PC.

Based on these considerations, we defined an initial ILHSI structure, focus areas, governance plan, and goals for the first 2 to 3 years. We purposely focused on a limited structure and objectives to establish feasibility and early success, instead of a full implementation of the LHS concept. As a result, we made two major choices: (a) While the ultimate vision for the ILSHI includes multiple, state‐wide partners, we started with a phased approach that focused on one health system and other early stakeholders, as the first step. Other stakeholder‐partners, such as patients and payers, and additional healthcare system‐partners are slated for future inclusion. This mirrors our experience with the establishment of the INPC in the late 80s, which started with a limited number of partners before growing into a state‐wide system. (b) We reserved planning for long‐term aspects such as sustainability; large‐scale evaluation of the impact on outcomes, quality of life and costs; and organizational culture change toward the LHS philosophy once initial feasibility of the effort had been established.

2.3. Implementation activities

Projects: To assist Executive Committee members in evaluating potential LHS projects, we developed a Project Assessment Tool, which categorized projects by their area of focus, potential impact, fit with LHS goals, and key stakeholders. Projects were scored according to scalability, development complexity, implementation complexity, impact timeline, allocated and required resources, and direct care impacts. Those project scores informed discussions and subsequent project selection. Toward the end of this initial phase, program leadership met to discuss and summarize lessons‐learned and inform future phases of work for the ILHSI.

Training: Recognizing that successful development of an LHS requires engaged providers integrated throughout the healthcare system, we created or leveraged related educational programs to engage clinicians and researchers in LHS activities, strengthening the involvement of the IUSM and the FSPH. The programs use the distinctive strengths of two academic partners to expand the ILSHI and thus benefit all. The LHS Young Investigator Awards, funded by the US Agency for Healthcare Research and Quality, train faculty members to improve patient care and health system operations through the systematic generation, adoption, and application of evidence. ²⁷ Trainees in the Indiana Public and Population Health Informatics Training Program (T15) (funded by the National Library of Medicine with the FSPH as its academic home and co‐directed by co‐author T.S.) and RI's clinical informatics fellowship participate in ILHSI projects. We also intentionally engaged junior faculty from the Advanced Scholars Program for Internists in Research and Education (ASPIRE) at IUSM to help shape scholars' projects around LHS activities and provide active mentorship on ILHSI projects. ²⁸

3. RESULTS

The collaborative planning process resulted in definition of the key foci, goals, and strategies of the ILHSI and establishment of a governance structure that has been leading the initiative through its first 2 years. This process was crucial to operationalizing the LHS and establishing the basis for a viable long‐term collaborative enterprise. Several key points emerged from this early planning process.

First, although all participants considered the concept of an LHS relevant and important, none felt their institution had implemented it to a sufficient degree to merit being designated a full LHS. Second, a regional, and eventually, statewide approach was valued and welcomed even by individual health system representatives. Participants expected that the impact of an LHS implemented broadly would be far greater than one focused on only one or a few organizations. This approach was also expected to resonate with state government priorities for improving health. Third, the group identified many ongoing quality improvement projects across the represented health systems, but also developed an appreciation that the LHS approach was appropriate for that subset intended for systematic, data‐driven evaluation, and scaling across an enterprise. Fourth, RI was considered a unique partner due to its position as an independent organization with expertise in multiple disciplines and clinical, analytic, and information linkages relevant to developing LHSs. This recognition crystallized RI's role as the driver of the ILHSI enterprise in these early stages and informed the governance structure described below.

3.1. Key foci

Key ILHSI foci emerged from this initial phase in four areas deemed essential to its establishment and sustainability, including measurement of success:

Informatics and data. Informatics and data are foundational to the ability of the LHS to measure quality, effectiveness, efficiency, and change, as well as the ability to drive change. Both are important parts of the infrastructure and methodology without which operationalization of an LHS is impossible, and both are core strengths of RI.

Organizational culture. Organizational culture is important in facilitating and/or slowing the implementation and evolution of an LHS. Successful LHSs make the continuous cycle of improvement an integral part of their organizational culture, ¹¹ , ¹³ , ¹⁷ not just an episodic and haphazardly applied philosophy.

Healthcare improvement. Demonstrable advances in health outcomes and process improvement were seen as central in the journey toward a successful ILHSI. This focus also correlated well with the significant opportunities for healthcare improvement evidenced by the fact that Indiana ranks in the bottom quintile regarding major measures of health. ²⁹

Translation, dissemination, and sustainability. The RI innovates in multiple domains of healthcare, such as informatics and care models. Successful translation of that research to the clinical environment was seen as an important measure of the ILHSI's success. At the same time, implementation is expected to provide inputs for future research, resulting in a translational research cycle. Dissemination, both within large healthcare organizations such as IUH and among various healthcare organizations in Indiana, was an explicitly acknowledged challenge. Considering sustainability as soon as possible was felt to be essential to increasing the likelihood of creating significant, long‐term impact.

3.2. Goals and strategies

As a result of the planning process and previous work, the ILHSI PC defined a set of initial overall goals, accompanied by specific strategies to meet them (Table 2).

TABLE 2.

ILHSI goals and strategies

Goals	Strategies to meet goals
Overcome fragmentation and silos among current LHS‐related units and committees	Create a strong partnership of equals focused on LHS development among RI, health systems, academic partners and IHIE
“Learning from every patient” – learn from routine care	Leverage routine healthcare activities and encounters (eg, data collected through practice) to enable systematic learning and pragmatic evidence generation
“Scale and Spread”—systematically identify, test, scale	Conceptualize, implement, and evaluate research and quality improvement projects to increase patient safety, improve patient outcomes, reduce provider burden/improve provider engagement, improve clinical/system efficiency, and improve the experience of care
Improve care and satisfaction among key stakeholders	Engage and evaluate LHS activities among clinicians, patients and other key stakeholders to improve healthcare outcomes as well as the healthcare experience
Improve process, efficiency, outcomes, cost	Develop a joint surveillance and communications infrastructure that supports shared decision‐making to identify high‐priority projects that improve outcomes while lowering costs.
Accelerate discoveries and translation of evidence	Work with multi/interdisciplinary research and development perspectives from academic partners and the Indiana Clinical and Translational Sciences Institute (Indiana CTSI) to create an environment for partners to test innovations/approaches that support LHS development in domains including clinical/translational research, precision health, and population health improvements
Disseminate discoveries, experiences and best practices	Create generalizable solutions implemented/adapted to local, regional, national, and international LHS contexts, and disseminate via traditional academic approaches (eg, publications and presentations) as well as commercialization, as appropriate. Increase the impact of solutions and innovations through local diffusion, commercialization, open sourcing, publication, and market dissemination approaches.
Leverage LHS approaches as key differentiator	Communicate “internally” across health system and partner organizations, and “externally” across the healthcare market the impacts and improvements related to adopting an LHS approach across partners and the region

3.3. Governance structure

Our initial governance structure was designed to ensure continuous collaboration among the ILHSI partners, while supporting efficient and effective decision‐making and operations. It consists of three components:

Executive Steering Committee (ESC). Having evolved from the PC, the ESC consists of representatives from executive and clinical management at IUH as well as senior leaders, faculty, and staff at RI. The ESC was charged with providing strategic guidance and direction to the initiative. It articulates LHSs priorities at the IUH system level, identifies relevant ongoing/past IUH initiatives, and creates alignments with RI strategic directions and capabilities. It identifies necessary resources both at IUH and RI, and directs their general allocation, with a particular focus on long‐term rollout and sustainability. It defines desired outcomes and metrics for the initiative, and reviews performance against these metrics.

Operations Committee (OC). The OC consists of operational administrative and clinical leaders, both within and across facilities in the IUH system, as well as RI faculty and staff. This committee is tasked with overseeing ongoing ILHSI projects.

ILHSI Program Team (PT). Composed of RI faculty and staff, as well as selected members of IUH (including clinicians), the PT plans for and implements the operations of the ILHSI. It is the “workhorse” committee for the ILHSI.

The committees intentionally span partners, organizational levels, and clinical disciplines ranging from executive and clinical operational leadership to faculty investigators, frontline clinicians (physicians and nurses), and health system analysts. This structure promotes ongoing environmental scanning, so that promising projects and operational needs are effectively brought into the LHS environment, emerging health system initiatives can be leveraged, and potential threats to LHS activities are anticipated.

3.4. Initial projects

From a list of more than 40 possible projects identified and ranked by the ESC using the Project Assessment Tool, six projects were chosen as our initial ILHSI activities (Table 3). These projects are monitored and advised in the Operations Committee, and any barriers to project success are brought to the ESC for active problem‐solving and direction. Hands‐on management of the projects is provided by the Program Team.

TABLE 3.

Initial ILHSI projects span a range of domains, such as clinical decision support, organizational culture improvement, quality of care, social determinants of health and population health (*current focus projects)

Project (area of care improvement)	Description and status
Health Dart (clinical decision support)	Health Dart is an application that integrates selected, high‐value data from the Indiana Network for Patient Care (INPC) relevant to a patient's chief complaint directly with Cerner, IUH's electronic health record. Health Dart has been found to increase use of health information exchange information and has been rolled out across all 15 EDs in the IUH system.
AHRQ Culture of Patient Safety Survey analysis (organizational culture improvement)	This project developed automated methods for categorizing and analyzing free‐text responses to the AHRQ Culture of Patient Safety Survey. Qualitative text analysis tools were used to categorize free‐text comments as having high positive or negative emotional content, and to cross‐reference this content with themes including responses to patient safety reporting and managerial support for patient safety reporting.
Chronic obstructive pulmonary disease (COPD) care improvement (quality of care)	This project developed and evaluated a COPD Care Management Program piloted among Academic Medical Center (AHC) COPD patients with high medical complexity and health care utilization. Core activities for the ILHSI included successfully spearheading efforts to have discrete pulmonary function test results integrated into the electronic health record, testing phenotype algorithms for diagnostic accuracy, and assisting with evaluation development and implementation tracking.
Predicting early mortality among transferred patients (quality of care, patient‐centered care)*	This project engages IUH operational leaders with an ILHSI‐mentored junior faculty member in the IUSM to examine patterns and identify factors related to early mortality among patients transferred at the IUH Academic Medical Center campus. A predictive model of early in‐hospital mortality risk was developed; future work will involve examining implementation of the model to guide clinicians and patients in critical illness conversations and inform medical decisions.
Uppstroms (social determinants of health)*	This project applies advanced machine learning to clinical and social datasets to identify patients in need of a referral to a social service such as a social worker, case manager, dietitian or legal medical partnership. Uppstroms, Swedish for “upstream,” is grounded in population health and risk stratification perspectives. Uppstroms has been in production use at EH since 2017.
Cardiovascular disease family risk assessment (population health)*	The cardiovascular disease (CVD) family risk assessment project will leverage data in the Indiana Network for Patient Care to predict heart attack risk based on family history of atherosclerotic CVD. Its primary objective is to identify and preventively engage with individuals at high risk of heart attacks. Resulting algorithms are intended to be piloted in the IUH Employee Health Plan.

3.5. Training for future LHS leaders

We successfully engaged several trainees in LHS activities during this early period. As one example, we partnered with the ASPIRE training program to mentor a junior hospitalist to conduct a project developing a model for predicting early in‐hospital mortality among patients transferred in to the IU Academic Health Center. ³⁰ This project is being carried forward in a current implementation project to identify patients for whom early palliative care consultation is indicated, and to determine barriers and facilitators to the use of this model among hospitalists.

4. DISCUSSION

In its initial phase, the ILHSI has been successful in developing and operationalizing a trans‐organizational LHS that may serve as one model for other collaborative LHS initiatives nationwide. At present, there appears to be only one other initiative, the HSSC, ²² that has developed along lines similar to the ILHSI. Common to both the HSSC and ILHSI, a strong, collaborative data infrastructure appears to be a foundational element for LHS efforts. As with the HSSC, our work across organizations, compared to within a defined organizational structure, has come with inherent potential benefits and challenges.

4.1. Lessons learned

In reflecting on the early stages of the development of the ILHSI, we have identified several important lessons learned in the process. We focus here primarily on lessons concerned with collaboration among the partners of this multi‐organizational LHS.

First, it has been essential that representatives from each core partner were engaged early in pre‐implementation and subsequently in all aspects of the ILHSI's development, bringing together academic, research, and operational leadership at every stage of the effort. We have been constantly guided by the principle that each partner is foundational to the ILHSI's success, contributing critical functions, perspectives, and expertise. We had noted in our literature review and previous experience that successful intra‐organizational LHSs have achieved organizational success through vertical integration of LHS core activities. ⁸ , ¹² , ³¹ However, a single organization's ability to engage and incentivize LHS program goals from the executive to the frontline levels, and to minimize information silos, is a distinct advantage that we, as a multi‐organizational collaboration, did not have. One way we addressed this challenge is with vertical alignment across core partners in the primary ILHSI committees (described above). The Chronic Obstructive Pulmonary Disease (COPD) project (see Table 3) is one example. In it, an early gap in care was identified and a clinical project team put in place, but the ILHSI's involvement brought both technical assistance for crucial data acquisition and phenotyping and evaluation assistance to understand core barriers and facilitators to delivering high‐quality and efficient COPD care.

Second, to minimize information silos, we developed the structure and methods to scan for LHS‐related activities and projects, and select projects for targeted infrastructure, experiential, and evaluative support to improve our ability to learn from and generalize successful LHS activities. Doing so was another important lesson learned. The Culture of Patient Safety project (see Table 3) was a result of this environmental scan, in which an identified operational need was matched with faculty technical expertise, and resulting new data were fed back to the operational team for health system use. In this project, we went beyond the information provision function of the LHS to engage in a culture change feedback loop by engaging implementation experts with operational leaders.

Third, in this initial phase of the ILHSI, we also learned ways to optimize our prospects for success by selecting the most promising projects for implementation. Explicitly and transparently rating projects in the ESC based on implementation complexity, timeline, resources available and required, and potential direct care impact has proved to be an effective way to target ILHSI activities for maximal system impact. This activity also has had the unexpected effect of providing a list of ready project ideas in which junior research and clinical faculty members at the IUSM can partner with operational system leaders to conduct small tests of change that can benefit the LHS, faculty advancement, and patient care. In the early mortality project (Table 3), for instance, ILHSI investigators partnered with the ASPIRE program to conduct a project that addressed an immediate priority for IUH clinical leadership. ³⁰

Fourth, our experience has confirmed our initial expectation that the LHS must be data‐driven, a strength of the RI and an important aspect of the collaborative nature of the ILHSI. A major challenge facing collaborative LHSs is interoperability of data. ⁶ , ²² , ³² , ³³ , ³⁴ , ³⁵ In developing the ILHSI, we have drawn on foundational lessons‐learned and relationships built from the INPC, which aggregates data from healthcare systems across Indiana, serving both patient care and research. The INPC acts as a critical enabling factor for the ILHSI by providing an existing relationship and governance structure that supports inter‐organizational data‐sharing and collaboration with the IHIE (which maintains the INPC) as well as practical infrastructure for aggregating data from multiple systems in support of LHS activities. ²³ This experience and infrastructure are fundamental to a multi‐organizational LHS that addresses the reality of patients engaging with multiple healthcare systems. The aggregation of data from multiple sources within Health Dart ³⁶ and the population health management strategies being tested in the COPD Care Improvement project is examples of the inclusion of data from multiple systems to promote LHS patient‐focused rather than system‐focused care improvements.

Among the lessons learned are some limitations we have recognized in our current efforts to develop a multi‐organizational LHS. Our decision not to include patients in the initial development likely shaped our initiative in ways that offer future enhancement opportunities. IU Health has fairly robust ways of involving patients through Patient Family Advisory Councils (PFCAs) and IU Health Insiders. These groups have participated in or provided input to the revision of ED operations, Clinical Quality Councils, dress code policies and, recently, virtual care. Involving patients early in the development of the ILHSI may have allowed us to address important questions that would add to the generalizability and the reach of the LHS, including: How does an LHS learn from the patient experience? How can the qualitative input of patients help in interpreting quantitative results from data analysis? What kind of LHS projects are important to patients? And, finally, how does the LHS close the loop to inform the patients about what it is learning? We plan to include a formal mechanism for patient input in the next iteration of IHLSI development, considering community engagement rather than simply a patient advisory group from one specific health care system. We also recognize the limitation of variable engagement across the operational partners. This caused the IHLSI to bring its new structure and partnerships to bear in selected projects to demonstrate the possibilities of LHS activities to new leaders not present at the time of the pre‐implementation planning and to those less familiar with the potential benefits of the IHLSI to their overarching health system goals. Finally, we are too early in our development to have mastered lessons about sustainability, however, we are synthesizing the lessons learned to modify our next phase of development to maximize opportunities to sustain the LHS as detailed below in our description of next steps.

Finally, we did not have a challenge common when developing interorganizational collaborations such as the ILHSI: that of bringing fierce competitors together and having to establish a foundation of trust. This is not to say that competition among Indiana's healthcare organizations does not exist—it very much does. However, the trust fabric established by the longstanding collaborative relationships of health systems with RI and among each other, most impressively demonstrated by the community's longstanding investment in and engagement with the INPC, ensured that the seed of the IHLSI would fall on fertile ground. Clearly, future discussions must address issues of whether and how existing quality improvement efforts are integrated into the ILHSI; how the ILHSI can take on a sustainable life of its own as opposed to being driven by RI; and how issues of governance, membership, innovation, and funding are addressed.

4.2. Next steps

The initial development of the ILHSI has identified areas where further work is needed to fully realize the goals of a collaborative LHS. The Institute of Medicine includes in its LHS definition the engagement of patients and families across all elements of the system, ²⁷ an area the ILHSI expects to develop in its next phase. Another area of ongoing discussion and planning is incentivizing LHS activities across the partners and stakeholders to enhance sustainability of both individual projects and the LHS culture more broadly. For example, research and clinical funding for LHS projects in the VHA have been aligned with senior executive performance plans to stimulate ongoing collaborations between researchers, clinicians, and operational leaders to improve veteran care and outcomes. A model like this could be an effective way of further embedding and incentivizing LHS activities into operational planning.

Current plans for building on the initial phase of developing the ILHSI include the following.

Alignment with organizational priorities: Initially, projects were targets of opportunity, identified on a first‐come, first‐served basis. Over time, we learned that projects aligned with organizations' strategic priorities were more likely to be successful and have resources assigned to them. We also recognized that while there remains engagement and interest in the LHS concept, additional work remains to achieve full adoption of an LHS approach as a core strategic driver and differentiator across our health system partners. While we had, initially, good support from some parts of executive leadership, significant turnover, as well as the COVID‐19 pandemic, disrupted the continuity of engagement. Although RI has been the initial institutional driver of the ILHSI and will continue to be a key player along with the other core partners, our ultimate aim is for the ILHSI to be health system‐driven and embedded into the essential business of health systems statewide. This requires close integration of the LHS concept with health systems' organizational strategies and priorities.

Focus on formalized projects: The initial engagement on projects with IUH was often necessarily opportunistic and ad hoc, and the goals, outcomes, and value of these joint projects were not always equally understood by all participants. Future projects will be rigorously defined in terms of goals, scope, individual and mutual responsibilities, outcomes, and milestones. Establishing the IHLSI as a formal “external” evaluator for healthcare system projects is one important aspect of formalizing project selection and conduct. As with previous projects, all IHLSI projects must fill a well‐defined need of IUH and other operational partners.

Algorithm development and implementation as a common “project umbrella”: Partnerships tend to work best when competencies and needs are matched closely. RI has unique local competence in developing and successfully deploying algorithms in many contexts. IUH has a significant need for leveraging algorithms in clinically useful ways. Therefore, projects for the ILHSI's next phase will focus on partnering regarding algorithm development, implementation, and evaluation. We have identified three algorithm‐focused projects for the coming year: (a) Uppstroms, an algorithm to predict the need for social service referrals of patients ³⁷ , ³⁸ ; (b) predicting the risk of sudden heart attacks by constructing family histories of heart disease based on INPC data; and (c) predicting in‐hospital mortality for inpatients in the IUH system to improve observed to expected mortality ratios and promote patient‐centric critical illness conversations.

Importantly, Uppstroms has already been in production at EH since 2017 ³⁹ and we are now working on adapting it for IUH. Conversely, we are in discussions with EH about adaptation and implementation of Health Dart, described in Table 3. These developments are important exemplars for advancing common interventions across different clinical partners. While the implementation of the same algorithm often differs from one setting to another, the fact that RI is leading these implementations is important for efficient and effective learning and generalization.

Generalizable technical solutions: Implementing technical solutions across different organizations typically faces several implementation challenges, such as different EHR platforms, configurations, and underlying data models. We have been using two strategies to enhance generalizability of our technical solutions. First, since 2016, RI has been making heavy use of the Fast Healthcare Interoperability Resources standard, which allows developers to create apps that are more portable than those created using vendor‐specific tools. Second, IHIE is the deployment partner for all projects that involve the INPC. IHIE is very capable of deploying solutions to a multi‐vendor, complex technical environment. This competence has helped us produce applications which are positioned for adoption by many organizations.

Engagement with frontline personnel and middle management: Our initial focus in engagement of health system leaders necessarily involved primarily IUH senior management in high‐level visioning and strategic development of ILHSI core structures and partnerships to ensure support and alignment with organizational goals (top‐down). However, we also purposely involved middle managers and some frontline personnel in project planning, prioritization, and execution steps (bottom‐up). Our experiences to date have validated this approach as partnering with middle managers and staff is key to translating LHS goals to deep, sustainable, value‐added progress. We also observed that IUH's organizational culture is such that internal needs and demands articulated by middle management drive much of its agenda, a core part of learning from the people “on the ground” that we want to foster and incorporate more formally into the ILHSI.

Consecutive short‐term “sprints” for ILHSI development: Our experience has shown that creating long‐term, multi‐year plans for initiatives such as the ILHSI is challenging. Much more promising is an approach using consecutive short‐term sprints to foster progress toward the long‐term vision, while reacting to developments in an adaptive manner.

Engagement of faculty and staff at RI more broadly: The initial involvement of RI faculty and staff with the ILHSI was, necessarily, limited to very few individuals. We are now considering how to involve RI and IUSM faculty and staff in various aspects of the ILHSI—for example, increasing involvement of junior faculty and fellows, more explicitly linking educational and LHS activities, and learning from successful clinical implementation projects led by partners in areas not currently fully embedded in the LHS. In doing so, we are leveraging our intellectual resources more effectively and efficiently while also training and engaging more clinicians and researchers in LHS‐focused work.

Implementation of full circle learning. Another measure of success for the ILHSI is implementing and sustaining LHS continuous learning cycles that iteratively improve and scale based upon the LHS model. ³ , ⁴⁰ , ⁴¹ , ⁴² After 2 years, the emergence of this cycle is evident. The most prominent example is an LHS‐driven solution we call Health Dart. ³⁶ Since initial development of this data‐driven app for improving emergency department care, ongoing evaluation and widespread implementation across sites has continued. Moreover, early evidence has led to extramural funding for a multi‐year, multi‐site evaluation study from AHRQ, demonstrating the LHS goal of ongoing research and care improvement, with plans for broadening to multiple sites and healthcare specialties across the State of Indiana. Our second example is the algorithm‐driven solution called Uppstroms, mentioned above. ³⁷ , ³⁸ After initial deployment and testing at one site (EH), we are now planning to implement and evaluate the approach at another health system (IUH) to test generalizability and scalability via an LHS approach.

Continued documentation, analysis, and reporting. A final area of continuing interest is the need to fully document and analyze the development of the ILHSI in its current and future stages, and to report those results to the healthcare research, practice, and policy communities in order that all may benefit from our experiences. Among these efforts is our interest in assessing how our experience can serve as a case study in the growing scholarship on maturity levels in LHSs, ⁴³ as well as continuing to publish studies of both individual projects and overall system development.

It may be argued that the lessons learned in developing the ILHSI would be hard to apply to other sites without comparable infrastructure and resources. Clearly, RI's history of open collaboration with local health systems; IHIE's competence in deploying technology into complex, multi‐vendor environments; and the presence of a large and comprehensive health information exchange facilitated some of our work. However, it should be recognized that developing multi‐organizational, healthcare market‐focused LHS initiatives will always have to take the local context and idiosyncrasies into consideration. In addition, many of our activities did not require new funding, but made use of existing resources within the system at large.

5. CONCLUSION

The ILHSI represents a new, promising model for consolidating the academic health tripartite mission and moving beyond the single organization context to a unified mission of improving health and health care through advancing, applying, and disseminating knowledge as it applies to real people navigating health care within and across complex systems. As the ILHSI continues to develop, it will serve as a unique laboratory and exemplar of the pragmatic collaboration and resulting promise of the interorganizational LHS as we move toward the goal of regional and national LHSs.

CONFLICT OF INTEREST

Dr Michele Saysana serves as the Board Chair for the Indianapolis Coalition for Patient Safety (ICPS), which is a paid position, but the money is paid to IU Health Physicians for her time. ICPS is a non‐profit entity that the Indianapolis hospitals serve as members; their mission is to bring the hospitals and health systems together to work together to improve patient safety.

Dr Randall Grout has received institutional grant funding from McKinsey and Company in the last 12 months.

Dr Linda Williams receives grant support as co‐Principal Investigator from VHA HSR&D for the Expanding Expertise through e‐Health Network Development (EXTEND) QUERI Program.

Schleyer T, Williams L, Gottlieb J, et al. The Indiana Learning Health System Initiative: Early experience developing a collaborative, regional learning health system. Learn Health Sys. 2021;5(3):e10281. 10.1002/lrh2.10281