A maturity grid assessment tool for learning networks

Abstract

Background

The vision of learning healthcare systems (LHSs) is attractive as a more effective model for health care services, but achieving the vision is complex. There is limited literature describing the processes needed to construct such multicomponent systems or to assess development.

Methods

We used the concept of a capability maturity matrix to describe the maturation of necessary infrastructure and processes to create learning networks (LNs), multisite collaborative LHSs that use an actor‐oriented network organizational architecture. We developed a network maturity grid (NMG) assessment tool by incorporating information from literature review, content theory from existing networks, and expert opinion to establish domains and components. We refined the maturity grid in response to feedback from network leadership teams. We followed NMG scores over time for nine LNs and plotted scores for each domain component with respect to SD for one participating network. We sought subjective feedback on the experience of applying the NMG to individual networks.

Results

LN leaders evaluated the scope, depth, and applicability of the NMG to their networks. Qualitative feedback from network leaders indicated that changes in NMG scores over time aligned with leaders' reports about growth in specific domains; changes in scores were consistent with network efforts to improve in various areas. Scores over time showed differences in maturation in the individual domains of each network. Scoring patterns, and SD for domain component scores, indicated consistency among LN leaders in some but not all aspects of network maturity. A case example from a participating network highlighted the value of the NMG in prompting strategic discussions about network development and demonstrated that the process of using the tool was itself valuable.

Conclusions

The capability maturity grid proposed here provides a framework to help those interested in creating Learning Health Networks plan and develop them over time.

Abbreviations

EHR

electronic health record

IT

information technology

LHS

learning health system

LN

learning network

NAM

National Academy of Medicine

NMG

network maturity grid

QI

quality improvement

1. INTRODUCTION

The learning healthcare system (LHS) was proposed in 2007 by the National Academy of Medicine (NAM) to promote improvement in health and health care delivery by integrating the health care activities of patients, families, clinicians, researchers, and healthcare systems leaders. ¹ The NAM envisions that a successful LHS would enable healthcare systems to produce the best quality evidence; promote ongoing collaborative efforts; drive the process of discovery as a natural outgrowth of patient care; and ensure innovation, quality, safety, and value in health care. ¹ , ² Current literature on LHSs includes descriptions of condition and disease‐oriented systems ³ , ⁴ and characterizations of LHSs in specific settings and health systems. ⁵ , ⁶ , ⁷ , ⁸ Prior literature also highlights essential elements of LHSs ranging from data and analytics to partnership and funding, ⁶ presents a model for implementation including phases from design to dissemination, ⁷ explores necessary attributes of national networks, ⁹ , ¹⁰ , ¹¹ considers the ethics ¹² , ¹³ and practical aspects of conducting research in LHSs, ¹⁴ , ¹⁵ , ¹⁶ , ¹⁷ and explores barriers to realizing the goals of LHSs. ¹⁸ However, an overall conceptual framework and practical guidance on the process and tangible steps to develop and mature new LHSs, is lacking. Without this, growth and development of LHSs will be slowed, because available examples may not generalize across populations, conditions, or settings.

We have described previously the use of a network organizational architecture to develop learning networks (LNs), one type of LHS comprised of multiple care sites that engages all stakeholders (patients, families, clinicians, researchers, health systems) to improve health. ⁹ , ¹⁶ , ¹⁹ , ²⁰ We have also demonstrated that this organizational form can be used to improve health outcomes for populations across health care organizations, while simultaneously supporting research. ⁹ , ²⁰ Network organizations are now widespread across many industries. They provide a flexible and adaptive means of organizing service delivery, using an “actor‐oriented” organizational architecture built on three central components: (a) “actors” such as patients, families, clinicians, and researchers who are motivated to collaborate toward a common goal, (b) a “commons” where actors create and share resources, and (c) processes, infrastructure, and policies that facilitate continual collaboration. ²¹

The infrastructure of a LN is a combination of social and technical processes and structures. We have used structured systems and system improvement approaches to develop and continuously improve such LNs in recent years. ²⁰ , ²² Applying systems improvement methods requires defining the processes that a network uses and assessing their degree of development. A process is a series of steps undertaken to achieve a specific objective. Process maturity refers to the degree to which a process is capable of achieving a specific objective or outcome in a predictable way. The concept of process or capability maturity is widespread in other industries as a means of managing and improving organizational development. For example, the Capability Maturity Model Integration is used to guide and assess software development along five process levels, with the highest level being the ideal state. ²³

The literature describes two types of capability frameworks: a maturity model ²⁴ and a maturity grid. ²⁵ A maturity model identifies best practices and uses a yes/no questionnaire or checklist to assess performance (usually by a third party) with the goal of certification. A maturity grid identifies the characteristics that a process should have for high performance and is structured around a matrix with the levels of maturity in each cell. The grid is usually a less complex diagnostic process than a model.

Here, we propose a process maturity assessment tool, based on the concept of the capability maturity grid and designed for improvement networks, that incorporates concepts from LHSs. This tool may be used to generate self‐assessments and enhance understanding of existing networks. We describe the necessary components and processes needed for LN operation and strategic planning. Additionally, we outline a case example describing the experience of network leaders applying this grid to a current, evolving LN. Finally, we describe our learnings from applying this tool to LNs over time.

2. METHODS

2.1. Subjects

Nine LNs participated in the development and application of the NMG: the Autism Learning Health Network (ALHN), ²⁶ the Cystic Fibrosis Learning Network (CFLN), ²⁷ the Pediatric Rheumatology Care & Outcomes Improvement Network (PR‐COIN), ²⁸ the National Pediatric Cardiology Quality Improvement Collaborative (NPC‐QIC), ²⁹ Solutions for Patient Safety (SPS), ³⁰ the ImproveCareNow Network, ³¹ the Ohio Perinatal Quality Collaborative (OPQC), ³² the Improving Renal Outcomes Collaborative (IROC), ³³ and the All Children Thrive Learning Network. ³⁴ Characteristics of these networks are displayed in Table 1.

TABLE 1.

Characteristics of learning networks participating in evaluation of the network maturity grid

Learning network	Date established	Age at time of initial maturity grid scoring (years)	Number of member sites	Dedicated network staff a	Patient population
ImproveCareNow Network	2007	10	109 pediatric care centers	11.5	Children and adolescents with inflammatory bowel disease
Ohio Perinatal Quality Collaborative	2008	9	105 maternity hospitals, 23 outpatient maternity care providers, 52 neonatal units, and 5 federally qualified health centers	10	Women and infants
Solutions for patient safety	2009	8	112 hospitals	15.5	Hospitalized children and children at risk for hospitalization
National Pediatric Cardiology Quality Improvement Collaborative	2009	8	63 pediatric cardiac centers	6.7	Infants born with single ventricle heart defects
Pediatric Rheumatology Care and Outcomes Improvement Network	2010	7	18 pediatric rheumatology centers	2.8	Children with rheumatologic conditions
Autism Speaks: Autism Treatment Network/Autism Intervention Research Network on Physical Health	2015	2	12 academic medical centers	2.6	Children with autism spectrum disorders and their families
Cystic Fibrosis Learning Network	2015	2	29 cystic fibrosis clinics	5	Individuals with cystic fibrosis
All Children Thrive Learning Network	2015	2	59	12	Infants and children through age 9 years
Improving Renal Outcomes Collaborative	2016	1	23 nephrology clinics	4	Patients with kidney disease and their families

^aReflects full time equivalents including staff dedicated to quality improvement, project management, event planning, communications, and network operations for respective networks.

2.2. Developing grid content and constructs

Our objective was to create a network maturity grid (NMG) that would serve as a tool for assessing the evolution of core network processes or functions and for ongoing planning. We drew from content outlined by the NAM for LHSs and reviewed the content theory (ie, key driver diagrams and system diagrams) for existing affiliated networks to identify major process domains common across networks. We performed a literature review of network organizational tools and approaches to identify additional major processes. ⁶ , ⁷ , ⁹ , ¹⁰ , ¹⁴ , ¹⁵ , ¹⁷ , ¹⁸ , ²² , ³⁵ , ³⁶ , ³⁷ , ³⁸ We engaged individuals with expertise in network design and management (GD, CF, SF, CL, PM, LP, MS) to identify and refine the major grid domains; suggest relevant content; and review, revise, and prioritize content. We incorporated the entirety of information obtained through the steps outlined above to establish the domains for our maturity grid. The six domains developed through this process were: systems of leadership, governance and management, quality improvement, community building and engagement, data and analytics, and research.

We considered each domain individually to identify a set of processes that would be necessary and sufficient to create a comprehensive description of the constructs within each domain. Our goal was to have an approximately equal number of components for each domain. For each component, we created five cells that detail behaviors for the component at five levels. A 5‐point rating scale is then applied to rate each construct. The scale represented a range of maturity from category 1, not started, to category 5 indicating a component had achieved an idealized state. We define and describe the six domains below briefly and in more detail in the Supporting Information.

2.3. Systems of leadership

This domain is defined as a set of methods to encourage a network to perform as a system. Activities in this domain focus on developing the purpose of the organization, viewing the organization as a system, measuring the system, obtaining information to improve the system, planning for, and managing improvement. Together, these activities form a system for the leaders of a network to focus their learning, planning for improvement, and actions toward achieving the network's mission.

Components of this system include development of the network's purpose, the use of diagrams to illustrate the interdependencies of processes in the network system, a dashboard of measures that can be tracked over time, defined strategic planning processes, and the use of formal improvement methods to manage improvement initiatives. This domain also includes several elements to related to financial performance and sustainability.

2.4. Governance and management

Governance and management is the way policies, processes, norms, and actions of a network are structured, sustained, regulated, and held accountable. This domain includes the processes for organizing, planning, directing, and controlling resources within a network with the overall aim of achieving its mission. The way the network interacts with other organizations is also considered here.

The components of this domain address oversight of network operations, including routine evaluations of network policies, the integration of external collaborators into a growing network, and the incorporation of stakeholders into the governance process. This domain also addresses policies related to protecting and sharing of data generated within a network, with specific attention given to the secure handling of personal health information.

2.5. Quality improvement

The mainstay of an improvement network, quality improvement consists of systematic and continuous actions that lead to measurable improvement in outcomes of health care services and the health status of the network's targeted patient groups. After developing improvement capability, quality improvement processes include planning for improvement, developing innovations and new ideas to improve health, executing improvement at specific network centers, and scaling‐up improvements throughout the network.

The quality improvement domain addresses the systematic use of quality improvement methods within networks to support achievement of network goals. This domain also includes the development and incorporation of quality improvement education for all network members. Other key processes include consistent sharing of network learnings in aggregate and according to individual centers.

2.6. Engagement and community building

Engagement and community building describe the structures and processes that enable all stakeholders—parents, patients, clinicians, researchers—to act on their inherent motivations and become involved in the network. This domain includes establishing a culture of collaboration and partnership in all network activities and increasing awareness of and opportunities to engage in the network; developing capabilities and processes to facilitate coproduction of improvement and to grow local improvement leadership; and a system to create and share information, knowledge, and knowhow to get what is needed, when it is needed, for improving health and the healthcare system.

This domain also addresses cultivation of a network culture that encourages and values participation of all stakeholders with an emphasis on patients. It includes the development of communication methods to facilitate ongoing community member involvement and the promotion of self‐organizing teams within networks. Other important components are processes to encourage collaboration of stakeholders such that groups with common goals may be readily identified.

2.7. Data and analytics

Data and analytics refer to the activities of a network to collect, validate, organize, and standardize data relevant to a network's mission and making these data available to all network stakeholders for clinical care, improvement, research and learning.

Several processes are required to ensure that various types of data, including patient‐level data (point of care, biospecimens, clinical assessments), population‐level data, and clinical quality measures are routinely and reliably collected within networks. The electronic health record (EHR) should integrate seamlessly with other network systems and capture data that can be readily extracted for analysis. The domain includes processes to support efficient, on‐demand analyses such that learnings may be readily incorporated into care practices and inform the continuum of care from acute clinical visits to chronic care monitoring. Other key components of this domain include processes to support research efforts including tracking patient recruitment and consent. Finally, activities that provide patients with ongoing access to personal, as well as population‐level information, are important to promote self‐management.

2.8. Research

Activities in this domain address the use of a variety of research methods to drive the generation of new knowledge related to the network's clinical focus and knowledge related to improvement of the network itself. This domain includes both qualitative and quantitative research with a goal to leverage the data, technical and social infrastructure.

Components of this domain address various types of research, including observational studies, n‐of‐1 studies, and clinical trials. Other scientific processes include providing training and support to researchers to understand how to work within a LN, and to maximize use of network resources. The domain includes prioritization of research goals and standardization of protocol development procedures.

2.9. Network maturity assessment and ratings

The first version of the maturity grid was prepared in 2014. Since then, leaders of all nine established learning networks, including quality improvement (QI) and project management staff, have provided critical review of the maturity grid. Members of the leadership team from each respective network examined domains, individual components of domains and descriptions of measures of maturity of each component. We continually refined the maturity grid components in response to feedback from network QI leaders and project management staff. These network leaders also rated their networks using the NMG yearly from 2017 to 2019 for the nine established networks.

2.10. Case study

We solicited subjective input from the leaders of the National Pediatric Cardiology Quality Improvement Collaborative to better understand the process of using the NMG. We developed a narrative describing their experience applying this capability process tool to the development of their network and learnings from using this tool.

2.11. Analysis

We sought input from all nine network leadership teams on the descriptions of structure, function, and processes of each LN to evaluate if the NMG addressed all aspects of the networks in the six domains. We examined change of the NMG scores over time and asked network leaders to identify if improvements were being made in areas where the scale demonstrated changes. We used spider diagrams to plot change over time in the yearly assessments for each domain for each LN, noting that some changes were made to the components each year. Finally, to understand the variation of assessments within a network, we plotted scores for each domain component, with respect to SD, for one participating network.

3. RESULTS

3.1. Network characteristics and network maturity grid scoring patterns

In Table 1, we detail the characteristics of each participating LN that applied the NMG to network development. LNs were of various ages and sizes and served unique populations.

Each network leadership team conducted a self‐assessment annually from 2017 to 2019. We noted variation across the nine LNs in the initial NMG scores as well as variation in how the NMG scores changed over time across networks (Figure 1). In qualitative feedback from all networks, change in network maturity scores over time aligned with network leaders' reports about growth in specific domains and was consistent with network efforts to improve in various areas. The scores documented differences in maturation in the individual domains of each network. In one participating network, evaluation of scoring patterns and SD for each domain component indicated consistency among LN leaders in some but not all aspects of network maturity (Figure 2).

FIGURE 1.

Network maturity grid scores from 2017, 2018, and 2019

FIGURE 2.

Average scores and SD overall (noted with horizontal and vertical lines), and for each component within the six domains of the network maturity grid (National Pediatric Cardiology Quality Improvement Collaborative, 2019)

3.2. Case example: National Pediatric Cardiology Quality Improvement Collaborative

The National Pediatric Cardiology Quality Improvement Collaborative (NPC‐QIC) used the NMG as a framework for the annual strategic planning process in 2019. In preparation for the strategic planning retreat, the network's project manager categorized the network's work within the domains of the NMG, in partnership with the network's Quality Improvement leader. Additionally, each subset of work under the domains was outlined in its current state and recommended state for the upcoming year (sustain, reduce, increase). The NMG was completed by the NPC‐QIC Executive Leadership team (clinicians and parents) and staff (project management and quality improvement) and the results were circulated to the team in advance of the retreat. During the retreat, the executive leadership team discussed the recommendations and prioritized the work for the upcoming year. The NMG served as the framework for the strategic planning process. Through the process, NPC‐QIC was able to build upon previous use of the NMG, have rich discussion, plan strategically important improvements and identify priorities and actions for the upcoming year.

4. DISCUSSION

The network maturity grid described here provides a tool that can be used by teams interested in developing learning networks to assess the maturity of processes across a range of social and technical domains related to achieving the LHS vision. It provides a structured method to gauge network progress and to support detailed and strategic discussions among leadership team members. Initial use of the grid across nine LNs indicated consistency between network leaders' interpretations of network development and maturity grid scores across domains. Variability in initial scoring, and the independent nature of growth across domains, suggest that NMG scores reflect areas of relative process maturity in LNs. Variability in the SD for component scores across domains highlights that areas of agreement and disagreement exist with regard to network leaders' impressions about maturity.

Maturity models, more so than grids, have been applied to other health‐related fields previously. Maturity models have been developed in healthcare information technology (IT) ³⁹ , ⁴⁰ and public health, ⁴¹ , ⁴² as well other specialized areas such as business development within health care. ⁴³ Within IT especially, multiple maturity models are available that vary with regard to origin, depth, and scope, and early models formed the foundation for the development of newer, novel, second generation models. ³⁹ Maturity grids, which differ in some ways from models, ²⁵ have not been widely applied in healthcare. However, in some instances, both terms are used to describe a given tool. ⁴⁴ The natural history of maturity models in health care IT, with early models prompting the development of revised models, suggests that our NMG may inform the development of next generation maturity grids. Subsequent maturity grids may differ in depth and scope and continue to evolve.

Our maturity grid is a tool that incorporates and builds upon previous guidance on LHS development and implementation, filling a notable gap in the literature. Key components for a LHS were proposed by Psek et al, including data and analytics, people and partnerships, patient and family engagement, ethics and oversight, evaluation and methodology, funding strategies, organization, prioritization, and deliverables. ⁶ Forrest et al described four essential aspects for developing LHSs: learning as a community, digital architecture, quality improvement, and rapid research. ⁹ Phases of development of a LHS have also been described, including scanning and surveillance, design, implementation, evaluation, adjustment, and dissemination. ⁷ Our NMG builds upon each of these works, as well as expert opinion and user experience, to specify key domains of LNs, the components with in each domain, and provide detailed information on the development process for each component.

The present study has limitations. Our sample of participating networks was limited to only nine networks that have used the LN framework. Though these networks represent a spectrum of network sizes, missions and populations of interest, future use by other networks will provide further feedback to inform refinement of the MG. Additionally, our NMG was developed by a relatively small group of experts including leaders from these nine networks.

We faced challenges in developing reliable descriptions of each domain and component to create a comprehensive and widely applicable framework. This task was especially complex as many components of our grid are likely to be interdependent, and the maturation of a network in one domain may be facilitated or impeded by relative strengths and weaknesses in other domains. Some aspects of learning networks are also inherently more difficult to assess than others. Networks differ with regard to priorities, patient populations and condition focus, and resources as well. Although we found the maturity grid applicable across a range of networks, there may be limitations in its applicability across all types of networks. Given the complexity and dynamic nature of LNs, more work is necessary to fully understand and detail the stages of the development process for LNs and refine our descriptors of maturity levels. This NMG may also benefit from refinement over time to address rapidly evolving factors such as integration of advancing technology in networks. Additional measures to address data security, and ensure appropriate data use, will be paramount. Finally, refining this NMG as we accrue experience over time will minimize the potential for variation we identified in scoring patterns.

We believe that the NMG will be useful for individuals interested in developing LNs that use a network organizational model. LNs are complex social and technical systems. By creating a process‐oriented framework to assess the maturity of a LN across domains and processes, this NMG has the potential to facilitate ongoing strategic planning, and may shape the culture of evolving networks by enabling improvement of network structures and processes. We hope this tool will provide leaders with a method to assess the strengths and weakness of their networks' current process maturity so that they can prioritize areas of improvement and track improvement over time. We anticipate that use of this NMG will facilitate the ongoing development of LNs in order to successfully produce outputs, such as new evidence, and improve results such as health outcomes and system efficiency. The ability to identify areas of agreement and disagreement among network leaders on components of each domain may also be invaluable in supporting the development of a common “systems” view within a network organization. Finally, this tool may provide leaders and designers with the ability to compare and contrast LNs to support learning from variation across networks.

5. CONCLUSION

The capability maturity grid that we have developed appears to be useful in assessing the development of LNs over time. We anticipate that this framework will continue to evolve as it is used in more diverse types of settings.

CONFLICT OF INTEREST

Peter A. Margolis, Michael Seid, and Carole Lannon are inventors of technology to support Learning Health System Networks that has been licensed to Hive Networks by Cincinnati Children's Hospital. Christine L. Schuler, Lloyd P. Provost, Sandra Fuller, David Purcell, and Christopher B. Forrest report no conflicts of interest.

Supporting information

Table S1. Systems of leadership domain.

Table S2. Components of the governance and management domain.

Table S3. Components of the quality improvement domain.

Table S4. Components of the engagement and community building domain.

Table S5. Components of the data and analytics domain.

Table S6. Components of the research domain.

Lannon C, Schuler CL, Seid M, et al. A maturity grid assessment tool for learning networks. Learn Health Sys. 2021;5:e10232. 10.1002/lrh2.10232