A framework for understanding, designing, developing and evaluating learning health systems

Tom Foley; Luke Vale

doi:10.1002/lrh2.10315

. 2022 May 20;7(1):e10315. doi: 10.1002/lrh2.10315

A framework for understanding, designing, developing and evaluating learning health systems

Tom Foley ^1,^✉, Luke Vale ²

PMCID: PMC9835047 PMID: 36654802

Abstract

Introduction

A Learning Health System is not a technical project. It is the evolution of an existing health system into one capable of learning from every patient. This paper outlines a recently published framework intended to aid the understanding, design, development and evaluation of Learning Health Systems.

Methods

This work extended an existing repository of Learning Health System evidence, adding five more workshops. The total was subjected to thematic analysis, yielding a framework of elements important to understanding, designing, developing and evaluating Learning Health Systems. Purposeful literature reviews were conducted on each element. The findings were revised following a review by a group of international experts.

Results

The resulting framework was arranged around four questions:

What is our rationale for developing a Learning Health System?

There can be many reasons for developing a Learning Health System. Understanding these will guide its development.
What sources of complexity exist at the system and the intervention level?

An understanding of complexity is central to making Learning Health Systems work. The non‐adoption, abandonment, scale‐up, spread and sustainability framework was utilised to help understand and manage it.
What strategic approaches to change do we need to consider?

A range of strategic issues must be addressed to enable successful change in a Learning Health System. These include, strategy, organisational structure, culture, workforce, implementation science, behaviour change, co‐design and evaluation.
What technical building blocks will we need?

A Learning Health System must capture data from practice, turn it into knowledge and apply it back into practice. There are many methods to achieve this and a range of platforms to help.

Discussion

The results form a framework for understanding, designing, developing and evaluating Learning Health Systems at any scale.

Conclusion

It is hoped that this framework will evolve with use and feedback.

Keywords: implementation science, informatics, learning health systems, learning healthcare systems, quality improvement

1. INTRODUCTION

The challenges facing healthcare are well rehearsed. ¹ Data and technology are often cited as the solution. However, health information technology often fails to deliver. ² The development and adoption of a Learning Health System has been proposed as a way of organising health systems and digital interventions to overcome previous failures.

A Learning Health System is not a new technical project within an existing health system, rather, it is the evolution of the existing system into one that is capable of learning from every patient who is treated.

The concept of a Learning Health System was introduced to healthcare in 2007 by the United States Institute of Medicine (IoM, now the National Academy of Medicine), ³ which later defined it as a system in which:

“science, informatics, incentives, and culture are aligned for continuous improvement and innovation, with best practices seamlessly embedded in the delivery process, [with] patients and families active participants in all elements, and new knowledge captured as an integral by‐product of the delivery experience.” ⁴

As awareness of the potential of Learning Health Systems has grown, a scientific ecosystem has developed around the Learning Health System concept, embodied by new university departments, ⁵ conferences ⁶ , ⁷ and an academic journal. ⁸

Several frameworks have been proposed to guide the development of Learning Health Systems. ⁹ , ¹⁰ , ¹¹ , ¹² , ¹³ , ¹⁴ While valuable, none of these has been universally applied in the development of new Learning Health Systems across settings and at different scales, covering social and technical factors. One has been successfully scaled across many new conditions internationally, ¹⁴ but does not provide an organisation‐level framework. Another is very widely cited and applies at any organisational scale. ¹⁵ However, the information available on the approach does not include detail on the full range of sociotechnical elements that often make the difference between success and failure. ⁹ Other frameworks may be too high‐level to be actionable, too specific to allow local tailoring, fail to account for complexity and sociotechnical challenges or are not accessible to the general reader.

This paper presents a new framework, visualised in Figure 1. This builds upon existing frameworks and aims to overcome some of the limitations noted above.

The framework described in Figure 1 was first outlined in a Health Foundation funded report in May 2021 ¹⁶ and has since been applied in a number of settings in order to aid the understanding, design, development and evaluation of Learning Health Systems. It is intended to be applicable at a local, regional, national and international scale. In the spirt of a learning system, this framework is a starting point that will evolve with use. It is hoped that this article will bring the framework to a new audience, who can participate in its evolution.

2. METHODS

This work builds on the 2015 report, The Potential of Learning Healthcare Systems. ¹⁷ The report established a web‐based repository of literature, interviews, expert workshops and other material related to Learning Health Systems. ¹⁸ Members of the public, patients and experts were invited to comment on the website material, which was then thematically analysed to produce the report. The website was updated with new findings as they emerged, even after publication of the report.

The interviews sought to explore the technical feasibility of Learning Health Systems and were conducted with international experts. Workshops focused on the, ethical, legal, regulatory, workforce planning, training, and economic implications of these potential developments. The interviews and workshops identified points around which there was consensus, disagreement or insufficient information to form an opinion.

For this current work, the structured review was updated with the addition of recently published systematic reviews on Learning Health Systems. ¹⁹ , ²⁰ Interviews and workshops conducted for the previous report were supplemented with five new expert workshops, covering topics that the authors identified as being important to the success of Learning Health Systems but not comprehensively covered in the literature of that field.

The first two workshops were face‐to‐face, but the final three were conducted online, due to Covid‐19 restrictions. Each workshop was chaired by a subject matter expert and drew upon a pre‐developed topic guide. This guide was developed by the authors and workshop chair. It was intended to target gaps and prompt discussion during the workshop. During each workshop, notes of the discussion were made, along with copies of any material produced. In addition, the online workshops were recorded. From these materials, a detailed summary of each workshop was produced.

Table 1 describes the topic of each workshop, the number of participants and their backgrounds. Each topic was focused on a broad area, relevant to the design and implementation of a Learning Health System. Findings were agreed with participants and added to the project website. ²¹

TABLE 1.

Outline of workshop topics, and background of workshop participants

Workshop topic	Number of participants	Background of participants
Mobilising Computable Biomedical Knowledge	52	Industry Academia Health service organisations
Non‐adoption, Abandonment, Scale‐up, Spread and Sustainability in Learning Health Systems	15	Implementation Science Academia Industry Health sciences
Evaluation in Learning Health Systems	9	Academia Health service organisations
Participatory Co‐Design in Learning Health Systems	15	Academia Health service organisations Charities Policy research groups
Platforms in Learning Health Systems	7	Health service organisations Industry

Open in a new tab

The interviews and products of the expert workshops, contained on the project website, along with the recent literature reviews were thematically analysed to construct a framework to aid the understanding, design, development and evaluation of Learning Health Systems.

Rapid purposeful literature reviews were then conducted for each element of the framework. These aimed to provide an introduction to the topic, rather than a systematic review. The results of these activities were presented in a draft report that was circulated for unstructured peer reviewed by a group of international Learning Health System experts, who had been selected on the basis of contribution to the field and willingness to participate (see acknowledgements). The framework was revised following their expert feedback.

No formal ethical approval for this work was sought. Potential workshop participants were identified from publicly available sources and asked if they would be willing to participate. Where workshops were recorded this was done with the verbal consent of all those participating.

3. RESULTS

Synopses of three of the expert workshops: (1) non‐adoption, abandonment, scale‐up, spread and sustainability in Learning Health Systems; (2) platforms in Learning Health Systems: and (3) evaluation in Learning Health Systems. These workshops were uploaded to the project website ²² alongside records of previous interviews and workshops that had been conducted since 2014. Videos of a fourth workshop (Mobilising Computable Biomedical Knowledge) have been made available ²³ , ²⁴ , ²⁵ , ²⁶ and documented through six independent peer reviewed articles. ²⁷ , ²⁸ , ²⁹ , ³⁰ , ³¹ , ³²

The framework draws on a wide range of literature. References in this paper include 45 relevant peer reviewed articles, 58 items of grey literature and 11 links to interviews or workshops conducted by the authors.

The thematic analysis identified a large number of issues that could influence the success of a Learning Health System. These were grouped into 30 topics that each applied at multiple scales, from local, to regional, national and international. These topics were then grouped under four questions that anyone developing a Learning Health System would need to ask (Box 1). These questions would be visited in an iterative, rather than a linear fashion. Findings related to each of these are provided in the following subsections.

BOX 1. Key questions to ask when developing a Learning Health System.

What is our rationale for developing a Learning Health System?
What sources of complexity exist at the system and the intervention level?
What strategic approaches to change do we need to consider?
What technical building blocks will we need?

3.1. What is our rationale for developing a Learning Health System?

Learning Health Systems come in many shapes and sizes. Their developers have differing objectives that influence how they are designed. Previous work ¹⁷ , ³³ has identified a variety of different rationales for developing a Learning Health System (this is illustrated in Figure 2):

To improve patient outcomes and experience
To provide better value healthcare
To identify and reduce unwarranted variation
To generate generalisable knowledge
To optimise the application of knowledge
To identify and track epidemiological phenomena
To maximise the benefits from existing technological investment
To enhance the education, training and performance of healthcare staff
Marketing

3.2. What sources of complexity exist at the system and the intervention level?

Healthcare is a complex adaptive system. ³⁴ It has many interacting elements, the whole is greater than the sum of the parts and outcomes are often unpredictable. ³⁵ As well as a clear rationale, an understanding of the complexity associated with health and Learning Health Systems is central to initiating, implementing, and sustaining Learning Health Systems in the real world. There have been several efforts to understand and manage such complexity, ² including the Non‐adoption, Abandonment, Scale‐up, Spread and Sustainability Framework for Health and Care Technology (NASSS). This combines 28 existing implementation frameworks and much primary research. ²

The workshop on complexity in the Learning Health System, concluded that the NASSS Framework was applicable to Learning Health Systems. ³⁶ It has therefore been incorporated into this framework (see Figure 3).

Sources of complexity in a Learning Health System (based on NASSS Framework)

NASSS highlights challenges to the success of an intervention across seven domains (see Figure 3). Each domain can be considered simple, complicated or complex. Too much complexity across too many domains makes the intervention more likely to fail ² , ³⁷ :

The condition: Some illnesses are more complex than others, in terms of aetiology, prognosis, sociocultural factors, comorbidities or treatments. It is more challenging to create systems that manage complex conditions.
The technology: Usability, dependability, maturity and requirements to integrate with other systems can all impact success.
The value proposition: The intervention must deliver value to the developer, the organisation, the patient, etc. or it will not be maintained or used.
The adopter system: Patients and carers may abandon a system that makes unreasonable demands on them. Clinicians may resist a system that threatens their job or professional identity.
The organisation: The organisation's capacity for innovation and its other priorities will influence its ability to develop a Learning Health System.
The wider context: Laws, regulation, politics, market conditions and other events can intervene to impact a Learning Health System.
Adapting over time: It is essential to the long‐term success of a system. All of the other six domains are subject to change.

The NASSS Framework is available in a series of self‐assessment tools called NASSS‐CAT (Complexity Assessment Tool). ³⁷ They can be applied to an entire Learning Health System or to individual interventions within a Learning Health System and are useful at the design, implementation and evaluation stages. ³⁶ They also include advice on reducing and responding to complexity.

3.3. What strategic approaches to change do we need to consider?

Continuous improvement within a Learning Health System requires change. The earlier report, ¹⁷ workshops and the literature, suggest a number of areas in which strategic approaches to change are required (see Figure 4).

Strategic approaches to change in a Learning Health System

Strategy: A Learning Health System requires a strategy and must be part of the broader strategy of its host organisation(s). The strategy is a set of inter‐related choices on how the organisation will deliver value. ³⁸ It can be an explicit document, outlining goals, priorities and actions, or it can be “a pattern in a stream of decisions” ‐ a strategic direction. ³⁹ The stakeholders must co‐design the strategy. There are many guides and tools that can help with this. ⁴⁰ , ⁴¹ , ⁴² , ⁴³

Organisational Structure: People and resources must be organised to deliver the strategy. This is sometimes called a target operating model (TOM) or an organisational architecture ⁴⁴ and has been explicitly addressed in various Learning Health Systems. ⁴⁵ Care is required to ensure that this approach does not become overly mechanistic, making unrealistic assumptions about the future.

Other approaches including, rapid prototyping, microstrategies, safe‐to‐fail experimentation ⁴⁶ and actor‐oriented learning networks, ⁴⁷ have been employed to more effectively meet the needs of Learning Health Systems. ⁴⁸ Actor‐oriented learning networks have been employed by ImproveCareNow, to create a Learning Health System for Inflammatory Bowel Disease ⁴⁹ and have since been spread more widely. ¹⁴

Culture: It has long been recognised as central to a Learning Health System. ⁵⁰ The organisation must be open to learning, innovation, sharing, research, etc. ⁵¹ Culture can be difficult to describe but can be thought of in 3 layers ⁵² :

Visible manifestations: Professional roles, physical layout of facilities, pathways of care, etc.
Shared ways of thinking: Values and beliefs that justify the visible manifestations.
Deeper shared assumptions: Unconscious underpinnings of daily practice.

These layers are learned and shaped through experience and distinct sub‐cultures can develop, such as among clinicians and managers. ⁵² There are quantitative and narrative‐based tools for assessing culture, but none is applicable in all situations. ⁵³ , ⁵⁴ Influencing culture is even more challenging, however, key targets and tools have been identified, with a focus on: vision, values, goals, compassion, learning, innovation, teamwork and collective leadership. ⁵⁵

Workforce: Developing a Learning Health System is a multidisciplinary endeavour. Patients, carers, clinicians, researchers, designers, strategists, analysts and other professionals have a role to play, depending on the particular implementation. The team could be made up from a mix of employed, contractors and consultants. Other staff may be employed by suppliers and platform providers. Leadership is especially important within such complicated organisational structures. ⁵⁶

As well as traditional responsibilities, clinicians will also need new skills and competencies:

Awareness of how the data that they record will be used. ⁵⁷
Interpreting the results of data analytics. ⁵⁸ , ⁵⁹ , ⁶⁰
Confidence with research as part of the clinical role. ⁵⁸
Ability and willingness to use data from non‐healthcare sources. ⁵⁷
Ability to use rapid and voluminous feedback on their own practice. ⁶¹

Many of these are represented in the Faculty of Clinical Informatics Competency Framework ²⁴ and take years to develop.

Implementation Strategy: A Learning Health System will often require an integrated package of evidence‐based interventions, designed to overcome barriers to successful implementation. This is known as an implementation strategy. ⁶²

Behaviour Change: A successful Learning Health System will require individuals and organisations to change what they are doing. This often requires purposeful intervention. There are many models that can be employed to guide this process. ⁶³ These often involve understanding what drives a given behaviour and then selecting techniques that can influence those drivers.

Co‐design: The complexity of a Learning Health System can only be understood by involving the stakeholders in its ongoing development. ⁶⁴ , ⁶⁵ Design principles have been developed, ⁶⁶ based on ISO 9241‐210, the international standard in human‐centred design for interactive systems. ⁶⁷ Libraries of open‐source co‐design tools are available. ⁶⁸ , ⁶⁹ , ⁷⁰ The choice of tools can be influenced by, the task, the type of knowledge sought and by the participants, ⁷¹ but there is limited evidence on how to choose the most appropriate tools. ⁷¹ , ⁷² It can be helpful to employ an overarching co‐design framework, ⁷³ , ⁷⁴ that offers a step‐by‐step guide. Crucially, it must not be a one‐off process, but rather a continuous part of a Learning Health System.

Appraisal and Evaluation: Guidance is available on appraising whether or not to build a Learning Health System, ⁷⁵ but, in practice, the decision can be reached by a multitude of formal and informal means. It might be driven by top‐down directive or bottom‐up interest. Pilots can be useful in testing hypothesised impacts. ⁷⁶

Once implemented, it is important to evaluate how effective, safe and cost effective a Learning Health System is. This allows an organisation to change direction and can enable others to avoid the same mistakes. There is guidance on evaluating complex systems, ⁷⁷ but Learning Health Systems can present particular challenges. A range of quantitative and qualitative methods might be appropriate (see Table 2). The mix will depend on the situation. Continuous improvement cycles, such as Plan‐Do‐Study‐Act, might be more feasible in some situations. ⁷⁶

TABLE 2.

Example analytic approaches within a Learning Health System ¹⁶

Analytic approach	Ideal use	Examples
Randomised trials	The most powerful knowledge generation methods in determining whether two treatments differ	Randomised controlled trials, ⁷⁸ A/B testing ⁷⁹
Quasi‐experimental analyses	When randomisation is challenging. For example, when a new policy is introduced nationally	Difference‐in‐differences analysis, interrupted time series, ⁸⁰ regression discontinuity, ⁸¹ causal inference, ⁸² artificial control groups, ⁸³ statistical process control ⁸⁴
Artificial Intelligence	Where large datasets exist, promising uses include, identifying undiagnosed disease, identifying new disease subtypes, predicting events or selecting treatments ⁸⁵	Deep learning, ⁸⁶ Machine learning, ⁸⁷ Penalised regression ⁸⁸
Descriptive statistics	When patterns can be identified without complex statistical techniques	Visualisation, process and value stream mapping ⁸⁹
Qualitative methods	When it is necessary to make sense of complex sociotechnical environments and to understand the why and how as well as the quantitative what	Interviews, focus groups, surveys and ethnography ⁹⁰

Open in a new tab

Maturity: In addition to evaluating outcomes, it is important to understand the degree to which processes within the Learning Health System are able to achieve those outcomes in a predictable way. ⁹¹ There are widely used maturity models for considering technical infrastructure ⁹² and there are process maturity assessment tools aimed at the complex, constantly changing sociotechnical aspects of a Learning Health System. ⁹¹

3.4. What technical building blocks will we need?

Once an organisation(s) understands why it is developing a Learning Health System and begins to appreciate the complexity and how change will be delivered, it can develop the technical building blocks required. Most Learning Health Systems build upon existing legacy systems. Locally, these legacy systems include electronic health records, data warehouses, decision support systems, learning communities and other sociotechnical systems. Nationally, these can include data sets, registries, guideline and policy development forums.

An early analysis of these systems, including strengths, gaps and opportunities for development, is therefore often a prerequisite to developing the technical building blocks. Resources are always limited and this analysis can help to prioritise where they can have most impact. Friedman and colleagues ⁹ have developed a generally accepted model, illustrating the core activities of a Learning Health System. This has been adapted to organise the technical building blocks within the new framework (Figure 5).

Technical building blocks of a Learning Health System

3.5. Learning communities

A learning community is a group of stakeholders (which may include patients, clinicians, managers, administrators, researchers, technical staff and others) who come together to achieve collective improvement. They are the centre of a Learning Health System. The learning community should drive the development and operation of a Learning Health System. They should implement a process for deciding what is to be studied and will also generate knowledge through reflection and shared experience. Such human communities are well placed to make sense of the complexity associated with a Learning Health System. The learning community may develop from an existing forum and there are helpful guides to operationalising a learning community, ⁹³ , ⁹⁴ but they should be tailored to each Learning Health System.

3.6. Practice to data

Once the learning community has decided what to study, the next step is to collect data that represents what is going on within the system. This requires several technical building blocks:

Data capture: Often from existing sources, such as the electronic health record, laboratories, remote monitoring tools or from technology (such as smartphones, wearables), social media, surveys, interviews, etc. ¹⁷ , ⁵⁸ , ⁹⁵ , ⁹⁶ , ⁹⁷ , ⁹⁸
Data Quality and Provenance: Routine data is collected as a by‐product of care, often without thought to their secondary uses. Quality can be variable. In order to rely on such data, it is important to consider the history of the data items and how they came to be in their current state. ⁹⁹
Data Storage and Access: Multi‐organisation Learning Health Systems must agree on whether data will be submitted to a centralised data store, such as England's Hospital Episode Statistics, ¹⁰⁰ or held locally in a distributed network, such as the FDA Sentinel Programme. ¹⁰¹ The former offers simplicity and economies of scale, while the later offers local control. Access to data can be disseminated to trusted parties (bringing data to the analysis) or users can analyse the data within a secure trusted research environment (bringing the analysis to the data).
Information Governance: Trust in a Learning Health System depends on data being collected, stored and shared within a strong and ethical framework. ¹⁰² In the United Kingdom and European Union, this is unpinned by the General Data Protection Regulation. ¹⁰³ In the United States, data sharing is regulated by the Health Insurance Portability Act. ¹⁰⁴
Interoperability: Learning Health Systems are often a network of networks that must be able to work with each other. This requires standards around, collection, the terminologies and classifications used, as well as the structure, transport and security of data. ¹⁷ , ²⁸ , ¹⁰⁵ With longitudinal data, standards for its representation can change over time, requiring complex mapping.

3.7. Data to knowledge

Data is of little value unless it can be turned into knowledge. Knowledge is the insight produced by analysing data and interpreting the product of that analysis. This could include a guideline, predictive model, calculator, etc. Organisations often collect huge amounts of data but fail to turn it into generalisable knowledge. ²⁸ Within a Learning Health System, knowledge can be generated by many statistical and deliberative methods, such as those shown in Table 2. These are described in more detail elsewhere. ¹⁶

Locally generated knowledge must be integrated with externally produced knowledge such as peer reviewed research or service evaluations conducted elsewhere. The objectives of the Learning Health System and the resources and infrastructure available will help to determine the methods and data which are most appropriate. This process should be led by the learning community.

3.8. Knowledge to practice

To improve health and care, it is not enough to generate knowledge. It must be applied. Traditionally, knowledge has been represented in books, journal articles, guidelines and protocols. It has been passed on through publications, education, training and word of mouth, before being unevenly applied.

These methods will continue to be essential within a Learning Health System, but there have also been efforts to standardise the representation of knowledge, ¹⁰⁶ , ¹⁰⁷ so that it can be delivered through decision support systems or used to assess whether real‐world care conforms to the evidence. This may greatly reduce the time taken for knowledge to be applied.

There are many challenges to mobilising knowledge within a Learning Health System ²³ , ³⁰ , ³¹ , ⁹⁵ :

Size of the task: The body of medical knowledge is huge.
Standards: Are required to make medical knowledge interoperable.
Ambiguity: Unlike traditional guidelines, computable guidelines require all background knowledge and common‐sense assumptions to be made explicit.
Inference: knowledge must be combined with data about the specific situation or patient to produce actionable outputs. This becomes even more challenging when there are conflicting knowledge sources, multiple valid interventions, multi‐morbidity, polypharmacy, etc.
Outcomes: Must be tracked to understand their magnitude and to who they accrue.
Safety: When such systems malfunction, large numbers of staff and patients can be affected. ¹⁰⁸ Decision support systems may be considered medical devices under some regulatory environments, such as the United Kingdom, ¹⁰⁹ , ¹¹⁰ , ¹¹¹ , ¹¹² and therefore require extensive development and validation.
Professionalism: Including clinical problem solving, craft skills and compassion, cannot be entirely replaced by computable knowledge.

For these reasons, progress on mobilising computable knowledge has often been slow.

In England, The National Institute for Health and Care Excellence (NICE) is planning a staged progression toward the goal of mobilising computable knowledge. NICE provides evidence‐based guidance to England's National Health System. They have assessed progress against the Agency for Healthcare Research and Quality's four levels of knowledge: narrative, semi‐structured, structured and executable. ²⁷ , ¹¹³ From this they have concluded that executable guidelines cannot be practically or safely reverse engineered from existing narrative guidelines. A new co‐production model of guideline development will be required, with different processes, methods, technologies and skills. ²⁷

3.9. Platforms

Much of the infrastructure required to build a Learning Health System is common across organisations. This can include IT, Electronic Record Systems, data transport, storage and processing, decision support and even governance arrangements and organisational structures. The development of platforms that provide this infrastructure as a service, is making it feasible and affordable for even small organisations to develop Learning Health Systems. ⁹⁵ The Mayo Clinic is pursuing this approach through its 10‐year Platform Strategic and Operating Plan. ¹¹⁴ They are purchasing platform services from other organisations, such as Google Cloud. They also hope to build platform services that may eventually be offered to other healthcare providers. These include a data platform for analytics, a virtual care platform for video consultations and remote monitoring and a remote diagnostic and management platform. ⁹⁵

The workshop on Platforms in the Learning Health System, agreed that “Platform Thinking” may reduce the technical demands on an organisation and enable economies of scale, but it requires business transformation and more collaboration with and trust for commercial and other partners. ⁹⁵

4. DISCUSSION

The elements of the framework, outlined in the previous section (Figures 2, 3, 4 and 5), have been developed through the analysis of interviews, workshops and purposeful literature reviews. They can be combined to create the overarching framework shown in Figure 1. Each ring represents elements of the answer to one of the questions in Box 1. The questions and thus the rings are related, but there is no direct mapping from the elements of one ring to those of the next.

The intention has been to create a framework that helps those wishing to understand, design, analyse or evaluate, real‐world Learning Health Systems, by highlighting elements that should not be overlooked. It is not intended to be a step‐by‐step, how‐to guide. The complexity of a Learning Health System would render such a guide ineffective because of the presence of non‐linear, unpredictable and constantly changing relationships within and between the elements, as described in the results section on complexity.

For the same reason, the framework does not explicitly position patients or clinicians, even though they are the most important actors in a Learning Health System. Instead, their presence is to be found in every ring of the framework, including within the learning community, at its centre.

In the design process, Figure 1 represents a framework for organising the key technical building blocks and platforms that will be required to deliver the aims of a proposed Learning Health System. It also highlights the potential sources of complexity and the strategic interventions that will be required to manage that complexity.

Organisations will often want to evolve their existing health system towards the goal of becoming a Learning Health System. In this scenario, the existing system can be mapped onto the framework, thus highlighting areas of strength, weakness or uncertainty. These can then be addressed systematically, using the tools identified.

Likewise, an organisation that has already embarked on a Learning Health System journey, can evaluate their progress in each element of the framework.

Learning Health Systems have been described as exhibiting fractal‐like properties, ¹¹⁵ meaning that the actions required to facilitate them are similar at whatever scale ‐ local, regional, national or international. The framework shown in Figure 1 shares that property. Each of the elements are relevant, regardless of the scale at which a Learning Health System is being developed.

4.1. Strengths, limitations and future work

This framework was developed based on the insights of a wide, but necessarily finite, group of subject experts and a series of purposeful literature reviews. The topics identified spanned 30 areas, each of which is a separate and evolving area of study. It was beyond the scope of this work to conduct systematic literature reviews on each of these areas. It is therefore possible that important insights have been missed or excluded due to space or capacity constraints.

To overcome this limitation, The Learning Healthcare Project have published a longer report, containing links to many additional resources. ¹⁶ Additionally, a collaboration of academics from Australia, the Netherlands, the United Kingdom and the United States has been formed to develop an online toolkit, based on the framework. Eventually, it should be possible for users around the world to add helpful Learning Health System tools. Analysis of these tools, patterns of usage within the toolkit and qualitative feedback will be used to evolve the framework's content and visual appearance, ensuring that it remains up‐to‐date, useful and becomes increasingly representative.

This paper began by identifying other frameworks and reasons why they might not have been adopted. It remains to be seen whether this framework will be any more widely adopted. It has been seven months since it was first published. Since then, the project website has had 9023 individual users from 149 countries (US 30%, UK 24%, Canada 10%, Australia 5% and China 4%). The lead author has been invited to present the framework to 16 national and regional forums. It has been used at NHS Digital to conduct a desk‐based training exercise on Learning Health Systems. Health Education England have embarked on a process to review their activity against the framework. The lead author has also applied the framework to assess the potential for a national Learning Health System initiative in England.

Early feedback from the above activities has resulted in some evolution of the framework, between the initial publication and this paper:

The importance of Learning Communities has been elevated, moving them to the centre of Figure 1.
It has been recognised that while Figure 1 is a useful visual checklist, it is not easily interpreted without detailed textual description.
The framework has therefore become more focused on the four questions in Box 1.
Figure 1 has been removed from the prototype toolkit and replaced with the four questions.
The four questions have been reordered.
The third question has been refocused on change, rather than just strategy.

This evolution is expected to continue and a governance system will be established to coordinate regular updates.

5. CONCLUSION

Interest in Learning Health Systems has grown since the term was first coined in 2007. ³³ This has created demand for a scalable framework to guide their development. This paper outlines such a framework, covering the rationale for a Learning Health System, sources of complexity, approaches to change and its technical building blocks.

It is hoped that this framework will aid those designing, analysing or evaluating Learning Health Systems. The framework itself is not set in stone, but will be updated over time, based on usage and feedback from the community.

CONFLICT OF INTEREST

Prof Luke Vale has no conflicts of interest to report. Dr Tom Foley has been commissioned by Health Education England to apply the framework to their efforts to become part of a national Learning Health System.

ACKNOWLEDGEMENTS

Prof Leora Horwitz (NYU Langone) and Raghda Zahran (Newcastle University) were co‐authors on the long report ¹⁶ describing the framework and contributed to multiple areas of its development, however they were not involved in the preparation or review of this manuscript and are therefore not listed as co‐authors.

We would like to thank The Health Foundation (Tom Hardie, Tim Horton and Will Warburton) for funding this project. We also benefited from logistical support and meeting rooms supplied by HDR UK (Alice Turnbull) and workshop support provided by Newcastle University Open Lab (Dr Jan Smeddinck). We are grateful for the ongoing support of Newcastle University and the Health Economics Group within the Population Health Sciences Institute, who host the Learning Healthcare Project. Dr Gregory Maniatopoulos helped with early project development. Graphics were created by Christopher Mathison.

The framework has been hugely improved by the input of reviewers: Sue Lacey Bryant (Health Education England), Prof Charles Friedman (University of Michigan), Tom Hardie (Health Foundation), Dr Anwar Hussain (Rhythm VC), Dr Philip Scott (University of Portsmouth) and Louise Wilson (Newcastle University). It has also been further improved by the Learning Health System Toolkit working group (Prof Jeffrey Braithwaite, Prof Robert Verheij, Dr Jess Moreno, Dr Paige McDonald, Sarah Gilman, Prof Kenneth Harwood, Dr Hub Wollersheim, Prof Philip Van der Wees, Josh Rubin).

The framework would not have been possible without the workshop attendees, who freely offered their time and knowledge to the project. A list of their names can be found in our project report. ¹⁶

This manuscript was much improved by the detailed feedback provided by our peer reviewers, the assistant editor and the editor.

Foley T, Vale L. A framework for understanding, designing, developing and evaluating learning health systems. Learn Health Sys. 2023;7(1):e10315. doi: 10.1002/lrh2.10315

Funding information Health Foundation

REFERENCES

1. Ham C. The challenges facing the NHS in England in 2021. BMJ. 2020;371:m4973. doi: 10.1136/bmj.m4973 [DOI] [PubMed] [Google Scholar]
2.Greenhalgh T, Wherton J, Papoutsi C, et al. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale‐up, spread, and sustainability of health and care technologies. J Med Internet Res. 2017;19(11):e367. https://www.jmir.org/2017/11/e367/ [DOI] [PMC free article] [PubMed] [Google Scholar]
3. McGinnis JM, Aisner D, Olsen LO. The Learning Healthcare System. Washington (DC), USA: National Academies Press; 2007. [Online]. Available: https://www.nap.edu/catalog/11903/the-learning-healthcare-system-workshop-summary [PubMed] [Google Scholar]
4. McGinnis JM, Stuckhardt L, Saunders R, Smith M. Best Care at Lower Cost: the Path to Continuously Learning Health Care in America. Washington (DC): National Academies Press; 2013. [PubMed] [Google Scholar]
5. University of Michigan . Revolutionizing Learning, Transforming Health. University of Michigan. https://medicine.umich.edu/dept/learning-health-sciences.
6. MCBK . Annual MCBK Public Meeting. MCBK. https://mobilizecbk.med.umich.edu/news-events/annual-meetings.
7. MCBK . Featured Events. Mobilizing Computable Biomedical Knowledge (MCBK). https://mobilizecbk.med.umich.edu/news-events.
8. Friedman C. Learning Health Systems. https://onlinelibrary.wiley.com/journal/23796146
9. University of Michigan . Our approach to Learning Health Sciences. https://medicine.umich.edu/dept/lhs/explore-learning-health-sciences/our-approach (accessed 25/10/21).
10. McLachlan S, Potts HWW, Dube K, et al. The Heimdall framework for supporting characterisation of learning health systems. J Innov Health Inf. 2018;25(2):77‐87. doi: 10.14236/jhi.v25i2.996 [DOI] [PubMed] [Google Scholar]
11. Menear M, Blanchette MA, Demers‐Payette O, Roy D. A framework for value‐creating learning health systems. Health Res Policy Syst. 2019;17(1):79. doi: 10.1186/s12961-019-0477-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
12. Partners M. Initiatives: Learning Health System. https://monashpartners.org.au/initiatives/learning-health-system/ (accessed 25/10/21).
13. Allen C, Coleman K, Mettert K, Lewis C, Westbrook E, Lozano P. A roadmap to operationalize and evaluate impact in a learning health system. Learn Health Syst. 2021;5(4):e10258. doi: 10.1002/lrh2.10258 [DOI] [PMC free article] [PubMed] [Google Scholar]
14. Children's C. Active Learning Networks. James M Anderson Center for Health Systems Excellence. https://www.cincinnatichildrens.org/research/divisions/j/anderson-center/learning-networks/active-emerging.
15. Flynn AJ, Friedman CP, Boisvert P, Landis‐Lewis Z, Lagoze C. The knowledge object reference ontology (KORO): a formalism to support management and sharing of computable biomedical knowledge for learning health systems. Michigan: Wiley Online Library; 2018. [Online]. Available: https://pubmed.ncbi.nlm.nih.gov/31245583/ [DOI] [PMC free article] [PubMed] [Google Scholar]
16. Foley TH, Zahran R. Realising the Potential of Learning Health Systems, Newcastle University; 2021. [Online]. Available: https://learninghealthcareproject.org/realising-the-potential-of-learning-health-systems/ (accessed 25/10/21).
17. Foley T, Fairmichael F. The Potential of Learning Healthcare Systems, The Learning Healthcare Project; 2015. [Online]. Available: https://learninghealthcareproject.org/wp-content/uploads/2015/11/LHS_Report_2015.pdf
18. L. H. Project . Learning Healthcare Systems. https://learninghealthcareproject.org (accessed 25/10/21).
19. Zurynski Y, Smith CL, Vedovi A, et al. Mapping the Learning Health System: A Scoping Review of Current Evidence. Sydney, Australia: Australian Institute of Health Innovation; 2020. [Online]. Available: https://research‐management.mq.edu.au/ws/portalfiles/portal/134364432/Publisher_version_open_access_.pdf [Google Scholar]
20. Enticott J, Johnson A, Teede H. Learning health systems using data to drive healthcare improvement and impact: a systematic review. BMC Health Serv Res. 2021;21(1):1‐16. [Online]. Available: https://bmchealthservres.biomedcentral.com/articles/10.1186/s12913-021-06215-8#Tab3 [DOI] [PMC free article] [PubMed] [Google Scholar]
21. L. H. Project . Project workshop repository. https://learninghealthcareproject.org/category/interviews/ (accessed 28/01/22).
22. L. H. Project . Learning Healthcare Project: Interviews. https://learninghealthcareproject.org/category/interviews/ (accessed 25/10/21).
23. Faculty of Clinical Informatics . David Wong talk ‐ UKworkshop on Mobilising computable biomedical knowledge (MCBK). Mobilising Computable Biomedical Knowledge (MCBK). https://www.youtube.com/watch?v=AZCl7xzXkKU&list=PLn7CfTWFFBSzTZtK4bTpo7Q2kBLjr0RUM&index=9.
24. Faculty of Clinical Informatics . Core competency framework for clinical Informaticians. The Faculty of Clinical Informatics. https://facultyofclinicalinformatics.org.uk/core-competency-framework.
25. Faculty of Clinical Informatics . Andrew Mitchell – UK Workshop on Mobilising Computable Biomedical Knowledge (MCBK). Mobilizing Computable Biomedical Knowledge (MCBK). https://www.youtube.com/watch?v=lIHJLRjUjRo&list=PLn7CfTWFFBSzTZtK4bTpo7Q2kBLjr0RUM&index=8&t=135s.
26. Faculty of Clinical Informatics . Nick Booth Introduction – UK Workshop on Mobilising Computable Biomedical Knowledge (MCBK). Faculty of Clinical Informatics. https://www.youtube.com/playlist?list=PLn7CfTWFFBSzTZtK4bTpo7Q2kBLjr0RUM.
27. Mitchell A. A NICE perspective on computable biomedical knowledge. BMJ Health Care Inf. 2020;27(2):1‐4. https://informatics.bmj.com/content/27/2/e100126 [DOI] [PMC free article] [PubMed] [Google Scholar]
28. Sebire N, Cake C, Morris A. HDR UK supporting mobilising computable biomedical knowledge in the UK. BMJ Health Care Inf. 2020;27(2):1‐3. https://informatics.bmj.com/content/27/2/e100122 [DOI] [PMC free article] [PubMed] [Google Scholar]
29. Wales A. Decision support for Scotland's health and social care: learning from an outcomes‐focused approach. BMJ Health Care Inf. 2020:1‐3. https://informatics.bmj.com/content/27/2/e100124 [DOI] [PMC free article] [PubMed] [Google Scholar]
30. Walsh K, Wroe C. Mobilising computable biomedical knowledge: challenges for clinical decision support from a medical knowledge provider. BMJ Health Care Inf. 2020;27(2):1‐2. https://informatics.bmj.com/content/27/2/e100121 [DOI] [PMC free article] [PubMed] [Google Scholar]
31. Wong D, Peek N. Does not compute: challenges and solutions in managing computable biomedical knowledge. BMJ Health Care Inf. 2020;27(2):1‐3. https://informatics.bmj.com/content/27/2/e100123 [DOI] [PMC free article] [PubMed] [Google Scholar]
32. Wyatt J, Scott P. Computable knowledge is the enemy of disease. BMJ Health Care Inf. 2020;27(2):1‐3. https://informatics.bmj.com/content/27/2/e100200 [DOI] [PMC free article] [PubMed] [Google Scholar]
33. Foley TJ, Vale L. What role for learning health systems in quality improvement within healthcare providers? Learn Health Syst. 2017;1(4):e10025. doi: 10.1002/lrh2.10025 [DOI] [PMC free article] [PubMed] [Google Scholar]
34. Braithwaite J. Changing how we think about healthcare improvement. BMJ. 2018;361:1‐5. https://www.bmj.com/content/361/bmj.k2014 [DOI] [PMC free article] [PubMed] [Google Scholar]
35. Snowden DJ, Boone ME. A Leader's framework for decision making. Harvard Business Review. https://hbr.org/2007/11/a-leaders-framework-for-decision-making. [PubMed]
36. Foley T. Nonadoption, abandonment, scale‐up, spread and sustainability in LHS workshop. The Learning Healthcare Project. https://learninghealthcareproject.org/wp-content/uploads/2021/04/NASSS-Workshop-Writeup-201020.pdf (accessed 01/12/21).
37.Greenhalgh T, Maylor H, Shaw S, et al. The NASSS‐CAT tools for understanding, guiding, monitoring, and researching technology implementation projects in Health and social care: protocol for an evaluation study in real‐world settings. JMIR Research Protoc. 2020;9(5):e16861. https://www.researchprotocols.org/2020/5/e16861/ [DOI] [PMC free article] [PubMed] [Google Scholar]
38. Lafley AG, Martin RL. Playing to Win: how Strategy Really Works. Boston, MA: Harvard Business Press; 2013. [Google Scholar]
39. Mintzberg H. Patterns in strategy formation. Int Stud Manag Organ. 1979;9(3):67‐86. https://www.tandfonline.com/doi/abs/10.1080/00208825.1979.11656272 [Google Scholar]
40. NHS Providers . Building a Digital strategy. https://nhsproviders.org/building-a-digital-strategy.
41. Porter ME. What is strategy? Harv Bus Rev. 1996;74(6):61‐78. https://hbr.org/1996/11/what-is-strategy 10158474 [Google Scholar]
42. Rumelt RP. Good strategy/bad strategy: the difference and why it matters. Emerald Insight. 2012;28(8). https://www.emerald.com/insight/content/doi/10.1108/sd.2012.05628haa.002/full/html [Google Scholar]
43. Sondhi RK. Total Strategy. London: BMC Global Services Publications; 2008. [Google Scholar]
44. StrategicCoffee . How to design a target operating model (TOM). StrategicCoffee. https://strategiccoffee.chriscfox.com/2012/12/how-to-design-target-operating-model-tom.html.
45. Bohmer R, Shand J, Allwood D, Wragg A, Mountford J. Learning systems: managing uncertainty in the new Normal of Covid‐19. NEJM Catalyst Innovations in Care Delivery. https://catalyst.nejm.org/doi/pdf/10.1056/CAT.20.0318.
46. Snowden D, Greenberg R. Cynefin ‐ weaving sense‐making into the fabric of our world. Cognitive Edge; 2020.
47.Britto MT, Fuller SC, Kaplan HC, et al. Using a network organisational architecture to support the development of learning healthcare systems. BMJ Qual Saf. 2018;27(11):937‐946. https://qualitysafety.bmj.com/content/27/11/937.abstract [DOI] [PMC free article] [PubMed] [Google Scholar]
48. IHI . Introduction to learning Health networks. IHI. https://ihi.webex.com/recordingservice/sites/ihi/recording/c1f65d0feafe4361a373e3feab417155/playback.
49. ImproveCareNow . ImproveCareNow https://www.improvecarenow.org/.
50. National Academy of Medicine . The learning Health system series. National Academy of Medicine. https://nam.edu/programs/value-science-driven-health-care/learning-health-system-series/.
51. Scobie S, Castle‐Clarke S. What can the NHS learn from learning health systems? Nuffieldtrust. https://www.nuffieldtrust.org.uk/files/2019-05/learning-health-systems-v3.pdf. [DOI] [PMC free article] [PubMed]
52. Mannion R, Davies H. Understanding organisational culture for healthcare quality improvement. BMJ. 2018;363:k4907. https://www.bmj.com/content/363/bmj.k4907 [DOI] [PMC free article] [PubMed] [Google Scholar]
53.Jung T, Scott T, Davies HT, et al. Instruments for exploring organizational culture: a review of the literature. Public Adm Rev. 2009;69(6):1087‐1096. https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1540-6210.2009.02066.x [Google Scholar]
54. Shirai B. Dave Snowden — how leaders change culture though small actions. Medium. https://medium.com/@brixen/dave-snowden-how-leaders-change-culture-though-small-actions-766cd2bf5128.
55. The King's Fund . Improving NHS culture. The King's Fund. https://www.kingsfund.org.uk/projects/culture.
56. Wachter RM. Making IT work: harnessing the power of Health information technology to improve Care in England. GOV.UK. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/550866/Wachter_Review_Accessible.pdf (accessed 2021).
57. Foley T, Fairmichael F. Workforce and training focus group. The Learning Healthcare Project. http://www.learninghealthcareproject.org/section/evidence/12/71/training-and-workforce-focus-group.
58. Foley T, Fairmichael F. Dr Paul Wallace interview. The Learning Healthcare Project. http://www.learninghealthcareproject.org/section/evidence/24/66/dr-paul-wallace-interview.
59. Foley T, Fairmichael F. Professor Charles Friedman interview. The Learning Healthcare Project. http://www.learninghealthcareproject.org/section/evidence/25/50/professor‐charles‐friedman‐%20interview.
60. Foley T, Fairmichael F. Dr Jeff Brown interview. The Learning Healthcare Project. http://www.learninghealthcareproject.org/publication/5/60/dr-jeff-brown-interview.
61. Foley T, Fairmichael F. Geisinger Health System Site Visit. The Learning Healthcare Project. http://www.learninghealthcareproject.org/section/evidence/34/63/site-visit-to-geisinger-health-system.
62. Bauer MS, Damschroder L, Hagedorn H, Smith J, Kilbourne AM. An introduction to implementation science for the non‐specialist. BMC Psychol. 2015;3(1):32. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4573926/ [DOI] [PMC free article] [PubMed] [Google Scholar]
63. Michie S, Stralen MMV, West R. The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci. 2011;6(1):42. https://implementationscience.biomedcentral.com/articles/10.1186/1748-5908-6-42 [DOI] [PMC free article] [PubMed] [Google Scholar]
64. Reponen S. Co‐design Framework. http://results.learning-layers.eu/.
65. Sanders EB‐N, Eva Brandt TB, A framework for organizing the tools and techniques of participatory design, Paper presented at, Proceedings of the 11th Biennial Participatory Design Conference, 2010, pp. 195–198. doi: 10.1145/1900441.1900476 [DOI]
66. NHS Digital . Design principles. NHS Digital. https://service-manual.nhs.uk/assets/nhs-design-principles.pdf.
67. ISO . Human‐centred design for interactive systems. ISO/TC 159/SC 4 Ergonomics of human‐system interaction. https://www.iso.org/standard/77520.html.
68. Service Design Tools . Service design tools. https://servicedesigntools.org/.
69. Design Kit . Design Kit. https://www.designkit.org/.
70. The Right Question . The right question. https://rightquestion.org/.
71.Batalden M, Batalden P, Margolis P, et al. Coproduction of healthcare service. BMJ Qual Saf. 2015;25(7):509‐517. https://qualitysafety.bmj.com/content/qhc/25/7/509.full.pdf [DOI] [PMC free article] [PubMed] [Google Scholar]
72. Vandekerckhove P. Generative participatory design methodology to develop electronic Health interventions: systematic literature review. JMIR Publ. 2020;22(4):e13780. https://www.jmir.org/2020/4/e13780/ [DOI] [PMC free article] [PubMed] [Google Scholar]
73. P. o. C. Foundation . EBCD: experience‐based co‐design toolkit. The Point of Care Foundation. https://www.pointofcarefoundation.org.uk/resource/experience-based-co-design-ebcd-toolkit/.
74. Digital Health . Co‐design in Health. Good Things Foundation: Digital Health Lab. https://digital-health-lab.org/codesign-in-health.
75. The Green Book: Central Government Guidance on Appraisal and Evaluation; 2020. [Online] Available: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/938046/The_Green_Book_2020.pdf [Google Scholar]
76. Foley T. Evaluation in LHS workshop. https://learninghealthcareproject.org/evaluation-in-lhs-workshop/.
77. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:1‐39. https://mrc.ukri.org/documents/pdf/complex-interventions-guidance/ [DOI] [PMC free article] [PubMed] [Google Scholar]
78. N. Digital . NHS DigiTrials. https://digital.nhs.uk/services/nhs-digitrials (accessed 25/10/21).
79. Horwitz LI, Kuznetsova M, Jones SA. Creating a learning Health system through rapid‐cycle, randomized testing. N Engl J Med. 2019;381(12):1175‐1179. http://sumry.s3.amazonaws.com/uc/a5d6aec2707df51d9ba9c671a1610f05_SoundingboardNEJM.pdf [DOI] [PubMed] [Google Scholar]
80. Penfold RB, Zhang F. Use of interrupted time series analysis in evaluating health care quality improvements. Acad Pediatr. 2013;13(6):S38‐S44. https://www.sciencedirect.com/science/article/abs/pii/S1876285913002106 [DOI] [PubMed] [Google Scholar]
81. Maciejewski ML, Basu A. Regression discontinuity design. JAMA. 2020;(4):381‐382. https://jamanetwork.com/journals/jama/article-abstract/2768094 [DOI] [PubMed] [Google Scholar]
82. Beran TN, Violato C. Structural equation modeling in medical research: a primer. BMC Res Notes. 2010;3(1):1‐10. https://bmcresnotes.biomedcentral.com/articles/10.1186/1756-0500-3-267 [DOI] [PMC free article] [PubMed] [Google Scholar]
83. Iacus SM, King G, Porro G. Causal inference without balance checking: coarsened exact matching. Polit Anal. 2012;20:1‐24. https://www.jstor.org/stable/41403736 [Google Scholar]
84. Pfadt A, Wheeler DJ. Using statistical process control to make data‐based clinical decisions. J Appl Behav Anal. 1995;28(3):349‐370. https://onlinelibrary.wiley.com/doi/10.1901/jaba.1995.28-349 [DOI] [PMC free article] [PubMed] [Google Scholar]
85. Liang G, Fan W, Luo H, Zhu X. The emerging roles of artificial intelligence in cancer drug development and precision therapy. Biomed Pharmacother. 2020;128:110255. https://www.sciencedirect.com/science/article/pii/S0753332220304479 [DOI] [PubMed] [Google Scholar]
86.Bejnordi BE, Veta M, Van Diest PJ, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA. 2017;318(22):2199‐2210. https://jamanetwork.com/journals/jama/fullarticle/2665774 [DOI] [PMC free article] [PubMed] [Google Scholar]
87.Seymour CW, Kennedy JN, Wang S, et al. Derivation, validation, and potential treatment implications of novel clinical phenotypes for sepsis. JAMA. 2019;321(20):2003‐2017. https://jamanetwork.com/journals/jama/article-abstract/2733996 [DOI] [PMC free article] [PubMed] [Google Scholar]
88.Pavlou M, Ambler G, Seaman SR, et al. How to develop a more accurate risk prediction model when there are few events. BMJ. 2015;351:h3868. https://www.bmj.com/content/351/bmj.h3868.full [DOI] [PMC free article] [PubMed] [Google Scholar]
89. Shostack GL. Designing services that deliver. Harv Bus Rev. 1984. https://hbr.org/1984/01/designing-services-that-deliver [Google Scholar]
90. Bradley EH, Curry LA, Devers KJ. Qualitative data analysis for health services research: developing taxonomy, themes, and theory. Health Serv Res. 2007;42(4):1758‐1772. https://pubmed.ncbi.nlm.nih.gov/17286625/ [DOI] [PMC free article] [PubMed] [Google Scholar]
91. Lannon C. A maturity grid assessment tool for learning networks, Learning Health Systems , 2020. https://onlinelibrary.wiley.com/doi/full/10.1002/lrh2.10232. [DOI] [PMC free article] [PubMed]
92. HIMSS Analysis . Adoption model for analytics maturity. HIMSS Analysis. https://www.himssanalytics.org/healthcare-provider-models/all.
93. Wilson L, Lowe T. The Learning Communities: Collective Improvement in Complex Environments. Newcastle Upon Tyne, UK: Newcastle University; 2019. [Google Scholar]
94. Ferguson L, Dibble M, Williams M, Ferraro J. Operationalizing a learning community for a learning health system: a practical guide, 2020. [Online]. Available: https://deepblue.lib.umich.edu/bitstream/handle/2027.42/163515/LearningCommunityPracticalGuide_FINAL.pdf?sequence=1&isAllowed=y
95. Foley T. Platforms in LHS workshop. The Learning Healthcare Project. www.learninghealthcareproject.org.
96. Inada‐Kim M, Donnelly T. Tech supported remote monitoring for COVID‐19. NHS England. https://www.nhsx.nhs.uk/blogs/tech-supported-remote-monitoring-covid-19/.
97. Muller JZ. The Tyranny of Metrics. New Jersey, USA: Princeton University Press; 2018. [Google Scholar]
98.Van der Merwe SE, Biggs R, Preiser R, et al. Making sense of complexity: using sensemaker as a research tool. MDPI. 2019;7(2):25. https://www.mdpi.com/2079-8954/7/2/25 [Google Scholar]
99. Xu S, Rogers T, Fairweather E, Glenn A, Curran J, Curcin V. Application of data provenance in healthcare analytics software: information visualisation of user activities. AMIA Summits Transl Sci Proc. 2018;2018:263‐272. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5961786/ [PMC free article] [PubMed] [Google Scholar]
100. NHS Digital . Data, insights, and statistics. NHS Digital. https://digital.nhs.uk/data-and-information/data-insights-and-statistics.
101. Sentinel System . Sentinel System. https://www.sentinelinitiative.org/about/how-sentinel-gets-its-data (accessed 2021).
102. NHSX . Information governance (IG). NHSX. https://www.nhsx.nhs.uk/information-governance/.
103. UK Gov . Data protection act. UK Public General Acts. https://www.legislation.gov.uk/ukpga/1998/29.
104. CDC . Health information privacy. Center for Disease Control. https://www.cdc.gov/phlp/publications/topic/healthinformationprivacy.html.
105. ONC , Connecting Health and Care for the Nation: a 10‐year vision to achieve an interoperable Health IT infrastructure, The Office of the National Coordinator for Health Information Technology, 2014. [Online]. Available: https://www.healthit.gov/sites/default/files/ONC10yearInteroperabilityConceptPaper.pdf
106. OpenClinical . OpenClinical. OpenClinical. http://www.openclinical.net.
107. MCBK . Mobilizing Computable Biomedical Knowledge community https://mobilizecbk.med.umich.edu/.
108. Digital Health . QRisk2 in TPP “fixed” but up to 270,000 patients affected. Digital Health. https://www.digitalhealth.net/2016/06/qrisk2-in-tpp-fixed-but-up-to-270000-patients-affected/.
109. NHS Digital . DCB0129. https://digital.nhs.uk/data-and-information/information-standards/information-standards-and-data-collections-including-extractions/publications-and-notifications/standards-and-collections/dcb0129-clinical-risk-management-its-application-in-the-manufacture-of-health-it-systems.
110. NHS Digital . DCP0160. NHS Digital. https://digital.nhs.uk/data-and-information/information-standards/information-standards-and-data-collections-including-extractions/publications-and-notifications/standards-and-collections/dcb0160-clinical-risk-management-its-application-in-the-deployment-and-use-of-health-it-systems.
111. UK Gov . Health and Social Care Act. Legislation UK. https://www.legislation.gov.uk/ukpga/2012/7/contents/enacted.
112. UK Gov . Guidance Notified bodies for medical devices. Gov UK. https://www.gov.uk/government/publications/notified-bodies-for-medical-devices/notified-bodies-for-medical-devices.
113. AHRQ . Improving quality of care through clinical decision support (CDS) takes a community. Agency for Healthcare Research and Quality. https://cds.ahrq.gov/cdsconnect/about.
114. Mayo Clinic . Platform revolution: curing more people, reaching more lives, anytime, anywhere. Mayo Clinic. https://newsnetwork.mayoclinic.org/discussion/platform-revolution-curing-more-people-reaching-more-lives-anytime-anywhere/.
115. Friedman CP, Rubin JC, Sullivan KJ. Toward an information infrastructure for Global Health improvement. Yearb Med Inform. 2017;26(1):16‐23. doi: 10.15265/IY-2017-004 [DOI] [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0001] 1. Ham C. The challenges facing the NHS in England in 2021. BMJ. 2020;371:m4973. doi: 10.1136/bmj.m4973 [DOI] [PubMed] [Google Scholar]

[lrh210315-bib-0002] 2.Greenhalgh T, Wherton J, Papoutsi C, et al. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale‐up, spread, and sustainability of health and care technologies. J Med Internet Res. 2017;19(11):e367. https://www.jmir.org/2017/11/e367/ [DOI] [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0003] 3. McGinnis JM, Aisner D, Olsen LO. The Learning Healthcare System. Washington (DC), USA: National Academies Press; 2007. [Online]. Available: https://www.nap.edu/catalog/11903/the-learning-healthcare-system-workshop-summary [PubMed] [Google Scholar]

[lrh210315-bib-0004] 4. McGinnis JM, Stuckhardt L, Saunders R, Smith M. Best Care at Lower Cost: the Path to Continuously Learning Health Care in America. Washington (DC): National Academies Press; 2013. [PubMed] [Google Scholar]

[lrh210315-bib-0005] 5. University of Michigan . Revolutionizing Learning, Transforming Health. University of Michigan. https://medicine.umich.edu/dept/learning-health-sciences.

[lrh210315-bib-0006] 6. MCBK . Annual MCBK Public Meeting. MCBK. https://mobilizecbk.med.umich.edu/news-events/annual-meetings.

[lrh210315-bib-0007] 7. MCBK . Featured Events. Mobilizing Computable Biomedical Knowledge (MCBK). https://mobilizecbk.med.umich.edu/news-events.

[lrh210315-bib-0008] 8. Friedman C. Learning Health Systems. https://onlinelibrary.wiley.com/journal/23796146

[lrh210315-bib-0009] 9. University of Michigan . Our approach to Learning Health Sciences. https://medicine.umich.edu/dept/lhs/explore-learning-health-sciences/our-approach (accessed 25/10/21).

[lrh210315-bib-0010] 10. McLachlan S, Potts HWW, Dube K, et al. The Heimdall framework for supporting characterisation of learning health systems. J Innov Health Inf. 2018;25(2):77‐87. doi: 10.14236/jhi.v25i2.996 [DOI] [PubMed] [Google Scholar]

[lrh210315-bib-0011] 11. Menear M, Blanchette MA, Demers‐Payette O, Roy D. A framework for value‐creating learning health systems. Health Res Policy Syst. 2019;17(1):79. doi: 10.1186/s12961-019-0477-3 [DOI] [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0012] 12. Partners M. Initiatives: Learning Health System. https://monashpartners.org.au/initiatives/learning-health-system/ (accessed 25/10/21).

[lrh210315-bib-0013] 13. Allen C, Coleman K, Mettert K, Lewis C, Westbrook E, Lozano P. A roadmap to operationalize and evaluate impact in a learning health system. Learn Health Syst. 2021;5(4):e10258. doi: 10.1002/lrh2.10258 [DOI] [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0014] 14. Children's C. Active Learning Networks. James M Anderson Center for Health Systems Excellence. https://www.cincinnatichildrens.org/research/divisions/j/anderson-center/learning-networks/active-emerging.

[lrh210315-bib-0015] 15. Flynn AJ, Friedman CP, Boisvert P, Landis‐Lewis Z, Lagoze C. The knowledge object reference ontology (KORO): a formalism to support management and sharing of computable biomedical knowledge for learning health systems. Michigan: Wiley Online Library; 2018. [Online]. Available: https://pubmed.ncbi.nlm.nih.gov/31245583/ [DOI] [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0016] 16. Foley TH, Zahran R. Realising the Potential of Learning Health Systems, Newcastle University; 2021. [Online]. Available: https://learninghealthcareproject.org/realising-the-potential-of-learning-health-systems/ (accessed 25/10/21).

[lrh210315-bib-0017] 17. Foley T, Fairmichael F. The Potential of Learning Healthcare Systems, The Learning Healthcare Project; 2015. [Online]. Available: https://learninghealthcareproject.org/wp-content/uploads/2015/11/LHS_Report_2015.pdf

[lrh210315-bib-0018] 18. L. H. Project . Learning Healthcare Systems. https://learninghealthcareproject.org (accessed 25/10/21).

[lrh210315-bib-0019] 19. Zurynski Y, Smith CL, Vedovi A, et al. Mapping the Learning Health System: A Scoping Review of Current Evidence. Sydney, Australia: Australian Institute of Health Innovation; 2020. [Online]. Available: https://research‐management.mq.edu.au/ws/portalfiles/portal/134364432/Publisher_version_open_access_.pdf [Google Scholar]

[lrh210315-bib-0020] 20. Enticott J, Johnson A, Teede H. Learning health systems using data to drive healthcare improvement and impact: a systematic review. BMC Health Serv Res. 2021;21(1):1‐16. [Online]. Available: https://bmchealthservres.biomedcentral.com/articles/10.1186/s12913-021-06215-8#Tab3 [DOI] [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0021] 21. L. H. Project . Project workshop repository. https://learninghealthcareproject.org/category/interviews/ (accessed 28/01/22).

[lrh210315-bib-0022] 22. L. H. Project . Learning Healthcare Project: Interviews. https://learninghealthcareproject.org/category/interviews/ (accessed 25/10/21).

[lrh210315-bib-0023] 23. Faculty of Clinical Informatics . David Wong talk ‐ UKworkshop on Mobilising computable biomedical knowledge (MCBK). Mobilising Computable Biomedical Knowledge (MCBK). https://www.youtube.com/watch?v=AZCl7xzXkKU&list=PLn7CfTWFFBSzTZtK4bTpo7Q2kBLjr0RUM&index=9.

[lrh210315-bib-0024] 24. Faculty of Clinical Informatics . Core competency framework for clinical Informaticians. The Faculty of Clinical Informatics. https://facultyofclinicalinformatics.org.uk/core-competency-framework.

[lrh210315-bib-0025] 25. Faculty of Clinical Informatics . Andrew Mitchell – UK Workshop on Mobilising Computable Biomedical Knowledge (MCBK). Mobilizing Computable Biomedical Knowledge (MCBK). https://www.youtube.com/watch?v=lIHJLRjUjRo&list=PLn7CfTWFFBSzTZtK4bTpo7Q2kBLjr0RUM&index=8&t=135s.

[lrh210315-bib-0026] 26. Faculty of Clinical Informatics . Nick Booth Introduction – UK Workshop on Mobilising Computable Biomedical Knowledge (MCBK). Faculty of Clinical Informatics. https://www.youtube.com/playlist?list=PLn7CfTWFFBSzTZtK4bTpo7Q2kBLjr0RUM.

[lrh210315-bib-0027] 27. Mitchell A. A NICE perspective on computable biomedical knowledge. BMJ Health Care Inf. 2020;27(2):1‐4. https://informatics.bmj.com/content/27/2/e100126 [DOI] [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0028] 28. Sebire N, Cake C, Morris A. HDR UK supporting mobilising computable biomedical knowledge in the UK. BMJ Health Care Inf. 2020;27(2):1‐3. https://informatics.bmj.com/content/27/2/e100122 [DOI] [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0029] 29. Wales A. Decision support for Scotland's health and social care: learning from an outcomes‐focused approach. BMJ Health Care Inf. 2020:1‐3. https://informatics.bmj.com/content/27/2/e100124 [DOI] [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0030] 30. Walsh K, Wroe C. Mobilising computable biomedical knowledge: challenges for clinical decision support from a medical knowledge provider. BMJ Health Care Inf. 2020;27(2):1‐2. https://informatics.bmj.com/content/27/2/e100121 [DOI] [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0031] 31. Wong D, Peek N. Does not compute: challenges and solutions in managing computable biomedical knowledge. BMJ Health Care Inf. 2020;27(2):1‐3. https://informatics.bmj.com/content/27/2/e100123 [DOI] [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0032] 32. Wyatt J, Scott P. Computable knowledge is the enemy of disease. BMJ Health Care Inf. 2020;27(2):1‐3. https://informatics.bmj.com/content/27/2/e100200 [DOI] [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0033] 33. Foley TJ, Vale L. What role for learning health systems in quality improvement within healthcare providers? Learn Health Syst. 2017;1(4):e10025. doi: 10.1002/lrh2.10025 [DOI] [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0034] 34. Braithwaite J. Changing how we think about healthcare improvement. BMJ. 2018;361:1‐5. https://www.bmj.com/content/361/bmj.k2014 [DOI] [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0035] 35. Snowden DJ, Boone ME. A Leader's framework for decision making. Harvard Business Review. https://hbr.org/2007/11/a-leaders-framework-for-decision-making. [PubMed]

[lrh210315-bib-0036] 36. Foley T. Nonadoption, abandonment, scale‐up, spread and sustainability in LHS workshop. The Learning Healthcare Project. https://learninghealthcareproject.org/wp-content/uploads/2021/04/NASSS-Workshop-Writeup-201020.pdf (accessed 01/12/21).

[lrh210315-bib-0037] 37.Greenhalgh T, Maylor H, Shaw S, et al. The NASSS‐CAT tools for understanding, guiding, monitoring, and researching technology implementation projects in Health and social care: protocol for an evaluation study in real‐world settings. JMIR Research Protoc. 2020;9(5):e16861. https://www.researchprotocols.org/2020/5/e16861/ [DOI] [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0038] 38. Lafley AG, Martin RL. Playing to Win: how Strategy Really Works. Boston, MA: Harvard Business Press; 2013. [Google Scholar]

[lrh210315-bib-0039] 39. Mintzberg H. Patterns in strategy formation. Int Stud Manag Organ. 1979;9(3):67‐86. https://www.tandfonline.com/doi/abs/10.1080/00208825.1979.11656272 [Google Scholar]

[lrh210315-bib-0040] 40. NHS Providers . Building a Digital strategy. https://nhsproviders.org/building-a-digital-strategy.

[lrh210315-bib-0041] 41. Porter ME. What is strategy? Harv Bus Rev. 1996;74(6):61‐78. https://hbr.org/1996/11/what-is-strategy 10158474 [Google Scholar]

[lrh210315-bib-0042] 42. Rumelt RP. Good strategy/bad strategy: the difference and why it matters. Emerald Insight. 2012;28(8). https://www.emerald.com/insight/content/doi/10.1108/sd.2012.05628haa.002/full/html [Google Scholar]

[lrh210315-bib-0043] 43. Sondhi RK. Total Strategy. London: BMC Global Services Publications; 2008. [Google Scholar]

[lrh210315-bib-0044] 44. StrategicCoffee . How to design a target operating model (TOM). StrategicCoffee. https://strategiccoffee.chriscfox.com/2012/12/how-to-design-target-operating-model-tom.html.

[lrh210315-bib-0045] 45. Bohmer R, Shand J, Allwood D, Wragg A, Mountford J. Learning systems: managing uncertainty in the new Normal of Covid‐19. NEJM Catalyst Innovations in Care Delivery. https://catalyst.nejm.org/doi/pdf/10.1056/CAT.20.0318.

[lrh210315-bib-0046] 46. Snowden D, Greenberg R. Cynefin ‐ weaving sense‐making into the fabric of our world. Cognitive Edge; 2020.

[lrh210315-bib-0047] 47.Britto MT, Fuller SC, Kaplan HC, et al. Using a network organisational architecture to support the development of learning healthcare systems. BMJ Qual Saf. 2018;27(11):937‐946. https://qualitysafety.bmj.com/content/27/11/937.abstract [DOI] [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0048] 48. IHI . Introduction to learning Health networks. IHI. https://ihi.webex.com/recordingservice/sites/ihi/recording/c1f65d0feafe4361a373e3feab417155/playback.

[lrh210315-bib-0049] 49. ImproveCareNow . ImproveCareNow https://www.improvecarenow.org/.

[lrh210315-bib-0050] 50. National Academy of Medicine . The learning Health system series. National Academy of Medicine. https://nam.edu/programs/value-science-driven-health-care/learning-health-system-series/.

[lrh210315-bib-0051] 51. Scobie S, Castle‐Clarke S. What can the NHS learn from learning health systems? Nuffieldtrust. https://www.nuffieldtrust.org.uk/files/2019-05/learning-health-systems-v3.pdf. [DOI] [PMC free article] [PubMed]

[lrh210315-bib-0052] 52. Mannion R, Davies H. Understanding organisational culture for healthcare quality improvement. BMJ. 2018;363:k4907. https://www.bmj.com/content/363/bmj.k4907 [DOI] [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0053] 53.Jung T, Scott T, Davies HT, et al. Instruments for exploring organizational culture: a review of the literature. Public Adm Rev. 2009;69(6):1087‐1096. https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1540-6210.2009.02066.x [Google Scholar]

[lrh210315-bib-0054] 54. Shirai B. Dave Snowden — how leaders change culture though small actions. Medium. https://medium.com/@brixen/dave-snowden-how-leaders-change-culture-though-small-actions-766cd2bf5128.

[lrh210315-bib-0055] 55. The King's Fund . Improving NHS culture. The King's Fund. https://www.kingsfund.org.uk/projects/culture.

[lrh210315-bib-0056] 56. Wachter RM. Making IT work: harnessing the power of Health information technology to improve Care in England. GOV.UK. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/550866/Wachter_Review_Accessible.pdf (accessed 2021).

[lrh210315-bib-0057] 57. Foley T, Fairmichael F. Workforce and training focus group. The Learning Healthcare Project. http://www.learninghealthcareproject.org/section/evidence/12/71/training-and-workforce-focus-group.

[lrh210315-bib-0058] 58. Foley T, Fairmichael F. Dr Paul Wallace interview. The Learning Healthcare Project. http://www.learninghealthcareproject.org/section/evidence/24/66/dr-paul-wallace-interview.

[lrh210315-bib-0059] 59. Foley T, Fairmichael F. Professor Charles Friedman interview. The Learning Healthcare Project. http://www.learninghealthcareproject.org/section/evidence/25/50/professor‐charles‐friedman‐%20interview.

[lrh210315-bib-0060] 60. Foley T, Fairmichael F. Dr Jeff Brown interview. The Learning Healthcare Project. http://www.learninghealthcareproject.org/publication/5/60/dr-jeff-brown-interview.

[lrh210315-bib-0061] 61. Foley T, Fairmichael F. Geisinger Health System Site Visit. The Learning Healthcare Project. http://www.learninghealthcareproject.org/section/evidence/34/63/site-visit-to-geisinger-health-system.

[lrh210315-bib-0062] 62. Bauer MS, Damschroder L, Hagedorn H, Smith J, Kilbourne AM. An introduction to implementation science for the non‐specialist. BMC Psychol. 2015;3(1):32. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4573926/ [DOI] [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0063] 63. Michie S, Stralen MMV, West R. The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci. 2011;6(1):42. https://implementationscience.biomedcentral.com/articles/10.1186/1748-5908-6-42 [DOI] [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0064] 64. Reponen S. Co‐design Framework. http://results.learning-layers.eu/.

[lrh210315-bib-0065] 65. Sanders EB‐N, Eva Brandt TB, A framework for organizing the tools and techniques of participatory design, Paper presented at, Proceedings of the 11th Biennial Participatory Design Conference, 2010, pp. 195–198. doi: 10.1145/1900441.1900476 [DOI]

[lrh210315-bib-0066] 66. NHS Digital . Design principles. NHS Digital. https://service-manual.nhs.uk/assets/nhs-design-principles.pdf.

[lrh210315-bib-0067] 67. ISO . Human‐centred design for interactive systems. ISO/TC 159/SC 4 Ergonomics of human‐system interaction. https://www.iso.org/standard/77520.html.

[lrh210315-bib-0068] 68. Service Design Tools . Service design tools. https://servicedesigntools.org/.

[lrh210315-bib-0069] 69. Design Kit . Design Kit. https://www.designkit.org/.

[lrh210315-bib-0070] 70. The Right Question . The right question. https://rightquestion.org/.

[lrh210315-bib-0071] 71.Batalden M, Batalden P, Margolis P, et al. Coproduction of healthcare service. BMJ Qual Saf. 2015;25(7):509‐517. https://qualitysafety.bmj.com/content/qhc/25/7/509.full.pdf [DOI] [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0072] 72. Vandekerckhove P. Generative participatory design methodology to develop electronic Health interventions: systematic literature review. JMIR Publ. 2020;22(4):e13780. https://www.jmir.org/2020/4/e13780/ [DOI] [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0073] 73. P. o. C. Foundation . EBCD: experience‐based co‐design toolkit. The Point of Care Foundation. https://www.pointofcarefoundation.org.uk/resource/experience-based-co-design-ebcd-toolkit/.

[lrh210315-bib-0074] 74. Digital Health . Co‐design in Health. Good Things Foundation: Digital Health Lab. https://digital-health-lab.org/codesign-in-health.

[lrh210315-bib-0075] 75. The Green Book: Central Government Guidance on Appraisal and Evaluation; 2020. [Online] Available: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/938046/The_Green_Book_2020.pdf [Google Scholar]

[lrh210315-bib-0076] 76. Foley T. Evaluation in LHS workshop. https://learninghealthcareproject.org/evaluation-in-lhs-workshop/.

[lrh210315-bib-0077] 77. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:1‐39. https://mrc.ukri.org/documents/pdf/complex-interventions-guidance/ [DOI] [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0078] 78. N. Digital . NHS DigiTrials. https://digital.nhs.uk/services/nhs-digitrials (accessed 25/10/21).

[lrh210315-bib-0079] 79. Horwitz LI, Kuznetsova M, Jones SA. Creating a learning Health system through rapid‐cycle, randomized testing. N Engl J Med. 2019;381(12):1175‐1179. http://sumry.s3.amazonaws.com/uc/a5d6aec2707df51d9ba9c671a1610f05_SoundingboardNEJM.pdf [DOI] [PubMed] [Google Scholar]

[lrh210315-bib-0080] 80. Penfold RB, Zhang F. Use of interrupted time series analysis in evaluating health care quality improvements. Acad Pediatr. 2013;13(6):S38‐S44. https://www.sciencedirect.com/science/article/abs/pii/S1876285913002106 [DOI] [PubMed] [Google Scholar]

[lrh210315-bib-0081] 81. Maciejewski ML, Basu A. Regression discontinuity design. JAMA. 2020;(4):381‐382. https://jamanetwork.com/journals/jama/article-abstract/2768094 [DOI] [PubMed] [Google Scholar]

[lrh210315-bib-0082] 82. Beran TN, Violato C. Structural equation modeling in medical research: a primer. BMC Res Notes. 2010;3(1):1‐10. https://bmcresnotes.biomedcentral.com/articles/10.1186/1756-0500-3-267 [DOI] [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0083] 83. Iacus SM, King G, Porro G. Causal inference without balance checking: coarsened exact matching. Polit Anal. 2012;20:1‐24. https://www.jstor.org/stable/41403736 [Google Scholar]

[lrh210315-bib-0084] 84. Pfadt A, Wheeler DJ. Using statistical process control to make data‐based clinical decisions. J Appl Behav Anal. 1995;28(3):349‐370. https://onlinelibrary.wiley.com/doi/10.1901/jaba.1995.28-349 [DOI] [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0085] 85. Liang G, Fan W, Luo H, Zhu X. The emerging roles of artificial intelligence in cancer drug development and precision therapy. Biomed Pharmacother. 2020;128:110255. https://www.sciencedirect.com/science/article/pii/S0753332220304479 [DOI] [PubMed] [Google Scholar]

[lrh210315-bib-0086] 86.Bejnordi BE, Veta M, Van Diest PJ, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA. 2017;318(22):2199‐2210. https://jamanetwork.com/journals/jama/fullarticle/2665774 [DOI] [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0087] 87.Seymour CW, Kennedy JN, Wang S, et al. Derivation, validation, and potential treatment implications of novel clinical phenotypes for sepsis. JAMA. 2019;321(20):2003‐2017. https://jamanetwork.com/journals/jama/article-abstract/2733996 [DOI] [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0088] 88.Pavlou M, Ambler G, Seaman SR, et al. How to develop a more accurate risk prediction model when there are few events. BMJ. 2015;351:h3868. https://www.bmj.com/content/351/bmj.h3868.full [DOI] [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0089] 89. Shostack GL. Designing services that deliver. Harv Bus Rev. 1984. https://hbr.org/1984/01/designing-services-that-deliver [Google Scholar]

[lrh210315-bib-0090] 90. Bradley EH, Curry LA, Devers KJ. Qualitative data analysis for health services research: developing taxonomy, themes, and theory. Health Serv Res. 2007;42(4):1758‐1772. https://pubmed.ncbi.nlm.nih.gov/17286625/ [DOI] [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0091] 91. Lannon C. A maturity grid assessment tool for learning networks, Learning Health Systems , 2020. https://onlinelibrary.wiley.com/doi/full/10.1002/lrh2.10232. [DOI] [PMC free article] [PubMed]

[lrh210315-bib-0092] 92. HIMSS Analysis . Adoption model for analytics maturity. HIMSS Analysis. https://www.himssanalytics.org/healthcare-provider-models/all.

[lrh210315-bib-0093] 93. Wilson L, Lowe T. The Learning Communities: Collective Improvement in Complex Environments. Newcastle Upon Tyne, UK: Newcastle University; 2019. [Google Scholar]

[lrh210315-bib-0094] 94. Ferguson L, Dibble M, Williams M, Ferraro J. Operationalizing a learning community for a learning health system: a practical guide, 2020. [Online]. Available: https://deepblue.lib.umich.edu/bitstream/handle/2027.42/163515/LearningCommunityPracticalGuide_FINAL.pdf?sequence=1&isAllowed=y

[lrh210315-bib-0095] 95. Foley T. Platforms in LHS workshop. The Learning Healthcare Project. www.learninghealthcareproject.org.

[lrh210315-bib-0096] 96. Inada‐Kim M, Donnelly T. Tech supported remote monitoring for COVID‐19. NHS England. https://www.nhsx.nhs.uk/blogs/tech-supported-remote-monitoring-covid-19/.

[lrh210315-bib-0097] 97. Muller JZ. The Tyranny of Metrics. New Jersey, USA: Princeton University Press; 2018. [Google Scholar]

[lrh210315-bib-0098] 98.Van der Merwe SE, Biggs R, Preiser R, et al. Making sense of complexity: using sensemaker as a research tool. MDPI. 2019;7(2):25. https://www.mdpi.com/2079-8954/7/2/25 [Google Scholar]

[lrh210315-bib-0099] 99. Xu S, Rogers T, Fairweather E, Glenn A, Curran J, Curcin V. Application of data provenance in healthcare analytics software: information visualisation of user activities. AMIA Summits Transl Sci Proc. 2018;2018:263‐272. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5961786/ [PMC free article] [PubMed] [Google Scholar]

[lrh210315-bib-0100] 100. NHS Digital . Data, insights, and statistics. NHS Digital. https://digital.nhs.uk/data-and-information/data-insights-and-statistics.

[lrh210315-bib-0101] 101. Sentinel System . Sentinel System. https://www.sentinelinitiative.org/about/how-sentinel-gets-its-data (accessed 2021).

[lrh210315-bib-0102] 102. NHSX . Information governance (IG). NHSX. https://www.nhsx.nhs.uk/information-governance/.

[lrh210315-bib-0103] 103. UK Gov . Data protection act. UK Public General Acts. https://www.legislation.gov.uk/ukpga/1998/29.

[lrh210315-bib-0104] 104. CDC . Health information privacy. Center for Disease Control. https://www.cdc.gov/phlp/publications/topic/healthinformationprivacy.html.

[lrh210315-bib-0105] 105. ONC , Connecting Health and Care for the Nation: a 10‐year vision to achieve an interoperable Health IT infrastructure, The Office of the National Coordinator for Health Information Technology, 2014. [Online]. Available: https://www.healthit.gov/sites/default/files/ONC10yearInteroperabilityConceptPaper.pdf

[lrh210315-bib-0106] 106. OpenClinical . OpenClinical. OpenClinical. http://www.openclinical.net.

[lrh210315-bib-0107] 107. MCBK . Mobilizing Computable Biomedical Knowledge community https://mobilizecbk.med.umich.edu/.

[lrh210315-bib-0108] 108. Digital Health . QRisk2 in TPP “fixed” but up to 270,000 patients affected. Digital Health. https://www.digitalhealth.net/2016/06/qrisk2-in-tpp-fixed-but-up-to-270000-patients-affected/.

[lrh210315-bib-0109] 109. NHS Digital . DCB0129. https://digital.nhs.uk/data-and-information/information-standards/information-standards-and-data-collections-including-extractions/publications-and-notifications/standards-and-collections/dcb0129-clinical-risk-management-its-application-in-the-manufacture-of-health-it-systems.

[lrh210315-bib-0110] 110. NHS Digital . DCP0160. NHS Digital. https://digital.nhs.uk/data-and-information/information-standards/information-standards-and-data-collections-including-extractions/publications-and-notifications/standards-and-collections/dcb0160-clinical-risk-management-its-application-in-the-deployment-and-use-of-health-it-systems.

[lrh210315-bib-0111] 111. UK Gov . Health and Social Care Act. Legislation UK. https://www.legislation.gov.uk/ukpga/2012/7/contents/enacted.

[lrh210315-bib-0112] 112. UK Gov . Guidance Notified bodies for medical devices. Gov UK. https://www.gov.uk/government/publications/notified-bodies-for-medical-devices/notified-bodies-for-medical-devices.

[lrh210315-bib-0113] 113. AHRQ . Improving quality of care through clinical decision support (CDS) takes a community. Agency for Healthcare Research and Quality. https://cds.ahrq.gov/cdsconnect/about.

[lrh210315-bib-0114] 114. Mayo Clinic . Platform revolution: curing more people, reaching more lives, anytime, anywhere. Mayo Clinic. https://newsnetwork.mayoclinic.org/discussion/platform-revolution-curing-more-people-reaching-more-lives-anytime-anywhere/.

[lrh210315-bib-0115] 115. Friedman CP, Rubin JC, Sullivan KJ. Toward an information infrastructure for Global Health improvement. Yearb Med Inform. 2017;26(1):16‐23. doi: 10.15265/IY-2017-004 [DOI] [PMC free article] [PubMed] [Google Scholar]

PERMALINK

A framework for understanding, designing, developing and evaluating learning health systems

Tom Foley

Luke Vale

Abstract

Introduction

Methods

Results

Discussion

Conclusion

1. INTRODUCTION

FIGURE 1.

2. METHODS

TABLE 1.

3. RESULTS

BOX 1. Key questions to ask when developing a Learning Health System.

3.1. What is our rationale for developing a Learning Health System?

FIGURE 2.

3.2. What sources of complexity exist at the system and the intervention level?

FIGURE 3.

3.3. What strategic approaches to change do we need to consider?

FIGURE 4.

TABLE 2.

3.4. What technical building blocks will we need?

FIGURE 5.

3.5. Learning communities

3.6. Practice to data

3.7. Data to knowledge

3.8. Knowledge to practice

3.9. Platforms

4. DISCUSSION

4.1. Strengths, limitations and future work

5. CONCLUSION

CONFLICT OF INTEREST

ACKNOWLEDGEMENTS

REFERENCES

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases