Implementing large-system, value-based healthcare initiatives: a realist study protocol for seven natural experiments

Abstract

Introduction

Value-based healthcare delivery models have emerged to address the unprecedented pressure on long-term health system performance and sustainability and to respond to the changing needs and expectations of patients. Implementing and scaling the benefits from these care delivery models to achieve large-system transformation are challenging and require consideration of complexity and context. Realist studies enable researchers to explore factors beyond ‘what works’ towards more nuanced understanding of ‘what tends to work for whom under which circumstances’. This research proposes a realist study of the implementation approach for seven large-system, value-based healthcare initiatives in New South Wales, Australia, to elucidate how different implementation strategies and processes stimulate the uptake, adoption, fidelity and adherence of initiatives to achieve sustainable impacts across a variety of contexts.

Methods and analysis

This exploratory, sequential, mixed methods realist study followed RAMESES II (Realist And Meta-narrative Evidence Syntheses: Evolving Standards) reporting standards for realist studies. Stage 1 will formulate initial programme theories from review of existing literature, analysis of programme documents and qualitative interviews with programme designers, implementation support staff and evaluators. Stage 2 envisages testing and refining these hypothesised programme theories through qualitative interviews with local hospital network staff running initiatives, and analyses of quantitative data from the programme evaluation, hospital administrative systems and an implementation outcome survey. Stage 3 proposes to produce generalisable middle-range theories by synthesising data from context–mechanism–outcome configurations across initiatives. Qualitative data will be analysed retroductively and quantitative data will be analysed to identify relationships between the implementation strategies and processes, and implementation and programme outcomes. Mixed methods triangulation will be performed.

Ethics and dissemination

Ethical approval has been granted by Macquarie University (Project ID 23816) and Hunter New England (Project ID 2020/ETH02186) Human Research Ethics Committees. The findings will be published in peer-reviewed journals. Results will be fed back to partner organisations and roundtable discussions with other health jurisdictions will be held, to share learnings.

Strengths and limitations of this study.

• Each of the value-based healthcare initiatives are based on evidence-informed clinical models of care, allowing our investigation to focus on understanding of the implementation strategies and processes that facilitate effective adoption and scaling.

• These seven natural experiments span many sites and patient conditions, creating the opportunity to explore diverse implementation strategies across a variety of settings.

• This research envisages a realist study, a methodological approach to provide a deep understanding of how complex implementation strategies and processes achieve a variety of outcomes for different populations, given varied circumstances and timelines.

• This study uses a mixed methods design to create optimal conditions for iterative exploration and rigour when developing programme implementation theories.

• Some assumptions may need to be made for the use of retrospective data to complement prospectively collected data, which potentially represents a limitation pending the direction and level of uncertainty surrounding these assumptions.

Introduction

Background

Globally, healthcare systems face unprecedented pressures on long-term performance and sustainability.^1–3 Substantial challenges include the ageing of populations, growing prevalence of chronic conditions, increased rates of hospital utilisation, rising costs of new medical technologies and medicines, and inefficient allocation of scarce resources.^4–6 Healthcare spending consumes an increasingly large proportion of government expenditure worldwide,⁷ without always delivering commensurate improvements in outcomes.⁸ This is evidenced by only around 60% of care aligning with current evidence, 30% considered low value and 10% resulting in harm.⁹ Further, patient expectations of healthcare are rising, and there is a developing understanding for how subjective patient outcomes, such as satisfaction and experience, can influence how people use the system and benefit from it.^{10 11} In response, governments and policy makers have turned their focus to optimising access, efficiency and quality of care through system-wide transformation.^12–15

Value-based healthcare

Value-based healthcare, a concept that has received substantial attention recently, is described as a shift away from ‘volume-oriented’ systems (activity as a primary measure of performance) towards those that additionally focus on the ’value’ of outcomes achieved.^{16 17} The core principles of value-based healthcare in New South Wales (NSW), Australia, expressly ‘considers what value means for patients, clinicians and the health system, and aims to provide health services that deliver value across four domains: improved health outcomes; improved experiences of receiving care; improved experiences of providing care; and better effectiveness and efficiency of care’.¹⁸ These principles are considered compatible with ensuring system sustainability, while keeping patient-centred care front of mind.¹⁸ How to best generate value is expressed differently throughout the academic literature,^19–21 and is unlikely to be entirely achieved by realignment of healthcare funding models alone. Efforts to date have largely focused on implementing evidence-based models that attempt to align delivery of the right care in the right setting at the right time, to reduce unwarranted practice variation and to promote patient-centred care.¹⁸

In NSW, Australia, the orientation towards value-based healthcare is focused on the ‘quadruple aim’ approach that prioritises resources towards improving population health, improving the experience of those receiving and providing care, and managing available resources and the per capita costs of care.²² There are several state-wide programmes designed to scale and support change through a range of system-wide approaches for targeted cohorts of people. Leading Better Value Care (LBVC) is one of those programmes and is positioned as a value-based healthcare programme for specific conditions that commenced public health system-wide implementation across over 100 health facilities in NSW from 2017 to 2018. The principles of the programme involve clinicians, networks and organisations implementing best-practice, value-based care, and treatment and management for chronic conditions.²³ Seven of the eight Tranche One LBVC initiatives have been identified as having a high potential to avoid unnecessary hospitalisation (which is a focus of this work) and are supported by multiple organisations, including the NSW Ministry of Health at the macro-level, and the Agency for Clinical Innovation (ACI), Bureau of Health Information and the Clinical Excellence Commission at the meso-level. Further details regarding the aims of each initiative are provided in table 1.

Table 1.

Description of the seven Tranche One LBVC initiatives

Initiative	Patient population	Description
Osteoarthritis chronic care programme	People with diagnosed osteoarthritis of the knee or hip	Aims to improve daily function and delay, avoid or improve recovery from knee or hip joint replacement surgery. The initiative involves expanding outpatient-based multidisciplinary conservative treatment options and support self-management strategies, including exercise, injury avoidance and encouragement of weight loss with drug therapies.
Osteoporosis refracture prevention	People aged 50 years and older who present to hospital with osteoporotic fracture	Aims to prevent refractures for people with osteoporosis by improving identification of conditions underlying minimal trauma fractures and streamlining case management processes. The initiative includes case management that supports patient access to medical consultations, community-based care and refracture education. Enhancing primary care management and fewer refractures is oriented towards reducing hospital admissions.
Chronic heart failure	People aged over 18 years, admitted with symptoms suggestive of chronic heart failure	Aims to reduce 28-day readmission and 30-day mortality by a focus on reducing unwarranted variation from best practice, enhance prevention, improve the management and mitigation of risks for people with chronic heart failure. These aims are to be achieved by improving early, accurate diagnosis, exacerbation management, transfer of care to multidisciplinary teams and palliative care. Multidisciplinary coordination of acute and primary care is leveraged to support self-management, including preventing acute exacerbation of chronic heart failure.
Chronic obstructive pulmonary disease	Acute admitted patients aged 40 years and older with COPD	Aims to reduce 28-day readmission and 30 day mortality by a focus on reducing unwarranted clinical variation and optimise lung function for people with COPD, with the goal of preventing deterioration that leads to acute episodes and hospital admission. The initiative involves patient education, chronic disease self-management and pulmonary rehabilitation with conventional drug therapies.
Inpatient management of diabetes mellitus	Acute admitted patients aged 16 years and older with diabetes requiring subcutaneous insulin management	Glycaemic instability puts inpatients with diabetes at greater risk of infections and other complications. This initiative aims to reduce the length of hospital admission for people with diabetes requiring subcutaneous insulin management by optimising glucose management. The initiative involves improving capacity to manage insulin and glucose among junior medical officers and general ward staff, and access to inpatient diabetes management teams, including safe transfers and standardised identification processes.
Diabetes high-risk foot service	People over 15 years of age with diabetic foot-related infections and/or ulcers of the foot or lower limb, including diabetes-related foot ulceration, infection and acute Charcot’s neuropathy	Aims to improve treatment and patient outcomes and reduce complications and associated hospitalisations for people with diabetic foot-related infections and/or ulcers of the foot or lower limb by enhancing equitable access to best-practice preventative care and management. The initiative involves outpatient-based high-risk foot services incorporating a multidisciplinary team approach and following treatment guidelines, and reducing state-wide variation in access to services.
Renal supportive care	People with chronic kidney disease or end-stage kidney disease deciding whether to pursue renal replacement therapies (RRT), conservatively managed without RRT, receiving RRT but experiencing symptoms impacting quality of life, and those withdrawing from dialysis	Aims to enhance patient (and carer) experience by supporting outpatient-based symptom management and palliative care for people with chronic and end-stage kidney disease. The initiative involves the Renal Supportive Care model of patient (and carer) support regardless of whether they embark on, or cease, RRT, backed by advanced and palliative care planning.

Source: Adapted and reproduced with permission from NSW Health, Leading Better Value Care, viewed 21 May 2020, <https://www.health.nsw.gov.au/Value/lbvc/Pages/default.aspx>.

COPD, chronic obstructive pulmonary disease.;.

While the initiatives focus on the organisation and delivery of care for specific patient cohorts, the overarching implementation strategies are based on guiding principles that span across the initiatives. The initiatives aim to create shared priorities between healthcare organisations, through an alignment framework that monitors care processes and outcomes against best-practice guidelines. NSW hospital networks (encompassing public hospitals, health institutions and services to defined geographical areas of the State, referred to as Local Health Districts) are responsible for designing local improvement plans that involve executive sponsorship, governance and leadership structures. Underpinning the implementation of the LBVC programme are resources provided by the supporting organisations in the form of project management, system-level enablers, funding for local implementation, an online collaboration portal, as part of a multilevel implementation package outlined in table 2. While the LBVC projects seek to focus on best clinical practice and value-based care,¹⁸ the important variations in context and how these interact with change mechanisms that drive implementation outcomes present unique challenges for the implementation of this system-wide programme at scale. This information is important in guiding how similar large-system projects can be scaled up and replicated in other local contexts.

Table 2.

Summary of the multilevel implementation package for the Leading Better Value Care programme

Macro-level policy and system drivers	A case for change and shared vision between health leaders across the state health system Promoting local adaptation and tailoring of programme implementation Data-driven monitoring and evaluation of progress and outcomes Provision of initial funding with the intention of local resource prioritisation to ensure sustainability
Meso-level implementation support agency working across health service organisations	Promotion of network weaving and extension of collaborations Providing local technical assistance Audit and provide feedback Create a learning collaborative Promoting adaptability and local tailoring
Hospital-level local implementation	Assessment of readiness and identification of barriers and facilitators Involvement of executive boards Creation/redesign/restructure of clinical teams Tailoring of implementation strategies Capturing and sharing local knowledge

Implementation and scaling of value-based care initiatives

The LBVC initiatives seek to accelerate best clinical practice and value-based care but the challenge lies in identifying how best to implement change at scale in complex adaptive healthcare systems. Healthcare systems are complex, adaptive, dynamic webs of vast numbers of interactions between healthcare professionals, administrators, support staff, patients, their families and a multitude of associated stakeholders.^24–26 There are few linear pathways, making implementation of any project a challenging endeavour. Along with the multitude of factors and interactions encountered when navigating complex healthcare systems, multiple aspects of implementation must also be considered. Much research effort is devoted to building the evidence base for particular models of care, in the form of clinical efficacy and effectiveness.²⁷ However, less focus is paid to improving implementation outcomes, distinct from service system outcomes and clinical treatment outcomes, that measure the effects of deliberate and purposive actions to implement new treatments, practices and services.²⁸ These measures focus on the acceptability, adoption, appropriateness, implementation costs, feasibility, fidelity, penetration and sustainment of implementing innovations (see online supplemental Appendix 1 for implementation outcome taxonomy), and are inherently linked to concepts of cost-effectiveness (as they are closely related to process measures used in economic evaluations and outcomes that are valued by patients). A focus on improving these outcomes is required to understand how the implementation and scaling of evidence-based models of care occurs. Therefore, it is imperative that we improve our understanding of local system dynamics, generative patterns of outcomes and conditional causal mechanisms for the implementation of large-system, value-based care transformation programmes, to inform health system improvement efforts. The evidence base for how to implement change will therefore not take the form of ‘what works’. Rather, it will be best placed to achieve broad understanding about ‘what tends to work, for whom, in what circumstances, how and why’.

Realist study

The level of complexity that is intrinsic to implementing large-system, value-based healthcare transformations presents a challenge to the use of traditional research methods. Healthcare operates in an open system, where regularities between events are not necessarily always able to be probabilistically specified or measured.²⁹ Therefore, it is important to apply methodologies that situate change as a conditional, contextually dependent and time-sensitive processes that occur across multilevel, nested and dynamic systems.³⁰ Realist studies, which take an approach grounded in the philosophy of scientific realism,³¹ are based on the assumption that programmes or initiatives work under certain conditions and are heavily influenced by the way that different stakeholders respond to them (see glossary of terms; table 3). A form of theory-driven research, realist studies seek to understand how a programme works, or is expected to work within specific contexts, and what conditions may hinder or promote successful outcomes.^32–34 A realist approach is ideal for complex, large-system health system implementation programmes,³⁵ as it acknowledges the non-linearity and context specificity of change process in complex adaptive health systems.^{36 37} This theory-driven method is increasingly being used for studying the implementation of complex interventions within health systems.³⁸

Table 3.

Glossary of key terms for implementation science, complexity theory and realist studies

Implementation science	The scientific study of processes to translate research evidence into practice, understanding what influences translational outcomes, and evaluating the adoption of interventions.26
Implementation outcome	The effects of deliberate actions to implement an innovation.28
Complexity	The behaviour embedded in highly composite systems or models of systems with large numbers of interacting components26; the complexity of a behaviour is equal to the length of its description.77
Adaptation	The capacity to adjust to internal and external circumstances; usually thought of in terms of modifying behaviours over time.26
Systems	A group of interacting elements that form a unified functional whole.26
Complex adaptive system	A dynamic, self-similar collective of interacting, adaptive agents and their artefacts.26
Realist study	A theory-driven approach based on a realist philosophy of science that is used to evaluate ‘what works, for whom, under what circumstances and how’ under the assumption that complex programmes and interventions work differently under certain circumstances.31 34 41
Programme theory	Description of what is supposed to be carried out in the implementation of programmes (theory of action) and how and why that is expected to work (theory of change).78 79
Context–mechanism–outcome configuration	Proposition-building set of possible explanatory relationships between the components of realist studies: (C) context or circumstances; (M) mechanisms or underlying social processes; (O) outcomes or results.80 81
Middle-range theory	Consist of limited sets of assumptions from which specific hypotheses are logically derived and confirmed by empirical investigation.82 These theories are considered more abstract and generalisable than ‘programme theories’ but do not constitute a ‘grand social theory’ themselves, instead they are considered adaptive and cumulative explanations.

Realist studies assert that programmes to implement healthcare improvements are ‘theories incarnate’,³⁴ that is, the roll out of programmes always has a theoretical underpinning, whether explicit or not. Therefore, it is important to make these theories explicit, from the perspectives of many actors. These programme theories are usually described as configurations between contexts (circumstances under which programme implementation works), mechanisms (generative causes of how implementation of programmes contribute to benefits) and outcomes (the intended consequences of programme implementation) (see figure 1 for context–mechanism–outcome (CMO) configuration example. Source: Adapted and reproduced from Dalkin et al.³⁹ Use of this image is supported by a Creative Commons License, http://creativecommons.org/licenses/by/4.0/). When focussing on the science around how a programme was implemented, rather than the programme itself, there are questions as to how ‘outcomes’ should be defined, whether to categorise process ‘outputs’ separately from outcomes, and which of these to include in a CMO configuration. Arguably, configurations of system and implementation efforts (context, actors, processes, content or object of the intervention, and implementation outcomes) could be first observed, then assessed on how they relate to endpoint patient and health outcomes. The process of theory refinement within a realist study would ideally inform the way these propositions are configured.

Figure 1.

Example context–mechanism–outcome configuration for why UK primary care practices placed fewer patients with non-cancer illness on their palliative care integrated care pathway register, compared with those with cancer illness.

Within a realist paradigm, context is more than a setting or place, rather it is anything which influences whether, how and why the mechanism will work. The epistemological assumption guiding realist studies is that programme implementation triggers ‘mechanisms’ of change differently, depending on the context. In realism, mechanisms are considered the ‘underlying entities, processes, or social structures which operate in particular contexts to generate outcomes of interest’,⁴⁰ and it is these mechanisms that trigger change rather than the programme implementation itself.¹⁵ Comparisons between CMO configurations can be used to formulate generalisable theoretical models (middle-range theories) for how and in what circumstances programme implementation achieves different outcomes. Delivering transferable models for how to implement large-system, value-based healthcare initiatives not only benefits ongoing application within their current contexts, but can inform how improvement initiatives can be scaled to other contexts, jurisdictions and settings.

Aims

The aim of this study is to undertake a realist study of the implementation of seven value-based healthcare initiatives. Specifically, we will develop and refine programme theories that unpack how varying strategies, contextual features and change mechanisms led to different implementation and programme outcomes. From this, we will build a nuanced model of the factors, conditions and levers that stimulate and inform the spread of local, multiagency, evidence-informed improvement projects into sustainable, nationwide impacts. These aims will be achieved by answering the research question: When implementing a large-system, state-wide value-based care programme to reduce unwarranted variation and improve patient-centred care, what tends to work, how, why, for whom, to what extent and in what circumstances, over what duration? The specific objectives of the study are to:

1. Identify the change strategies, processes and theorised mechanisms through which implementation of the seven value-based care initiatives influenced implementation and programme outcomes.

2. Investigate the impact of different contextual circumstances and agents on the mechanisms of initiative implementation over varying timelines.

3. Develop macro-level, meso-level and micro-level implementation models for successful value-based initiatives in healthcare, based on a mix of implementation strategies and mechanisms in specific contexts that achieve desired implementation outcomes.

Methods and analysis

Study design

This study will apply an exploratory, sequential, mixed methods realist study design, following the RAMESES II (Realist And Meta-narrative Evidence Syntheses: Evolving Standards) reporting standards for realist studies.⁴¹ Data will be collated from numerous sources to posit, test and refine hypothesised theories for how implementation strategies and processes lead to intended or unintended outcomes, depending on the influence of various contextual factors (eg, structure, relations, culture).^42–45 We will use both qualitative and quantitative data to uncover these patterns of outcomes in different contexts and to explicate the mechanisms for implementing value-based healthcare initiatives. Our iterative, three-stage model is depicted in figure 2 (Source: Adapted from Pawson et al,³⁴ Panmunthygala et al,⁴⁶ CMO: context–mechanism–outcome). Table 4 shows the three stages of the study and summarises each stage’s objective, the focal outcomes, sources of data and analysis approaches. Data collection for this study will commence in November 2020 and will be completed by July 2022.

Figure 2.

Three stages of the realist study and proposed process. CMO, context–mechanism–outcome configurations.

Table 4.

Relationship between data source, analysis approaches and research objectives

Stage	Data source	Data analysis	Objectives
1	Qualitative: Interviews with programme designers, supporting and evaluation staff from implementation support agencies Public and internal programme documents	Qualitative: Retroductive, theory-driven realist analysis Documentary analysis	(a) Identify the change strategies, processes and theorised mechanisms through which implementation of the seven value-based care initiatives influenced implementation and programme outcomes. (b) Investigate the impact of different contextual circumstances and agents on the mechanisms of initiative implementation over varying timelines.
2	Qualitative: Interviews with local hospital network staff Quantitative: Programme evaluation data Hospital administrative data Implementation outcome surveys	Qualitative: Retroductive, theory-driven realist analysis Quantitative: Descriptive and inferential statistical analysis
3	Data from stage 1, 2 and 3	Mixed methods: Realist triangulation	(c) Develop macro-level, meso-level and micro-level implementation models for successful value-based initiatives in healthcare, based on a mix of implementation strategies and mechanisms in specific contexts that achieve desired implementation outcomes.

Stage 1: identifying the initial programme theories

In stage 1, we will identify initial theories for how the value-based care initiatives are implemented into routine care. This stage will be informed by a realist synthesis of the literature around large-system hospital interventions (in preparation). Realist,^{47 48} exploratory semistructured interviews with programme implementers and evaluators, and analysis of documents from the macro-level and meso-level implementation support agencies will be used to map the actual process of implementation as it was done (work as done), compared with how it was planned (work as imagined).⁴⁹ This information will be used to refine existing programme logics and process models⁵⁰ for the implementation of value-based care initiatives. Additionally, relevant aspects of the social and environmental context will be collated and developed during the interviews and document review into determinant frameworks⁵⁰ to structure the analysis of the hypothesised CMO configurations. Through this mapping exercise, we will uncover the underlying system dynamics (relationships between agents, their attributes and rules of behaviour, network structure, feedback loops) that influence the implementation.

Stage 2: testing the hypothesised programme theories

During stage 2, focus will shift from developing theories, to testing theories and refining meta-narratives for the configuration of implementation support across initiatives. A mixed method approach using realist, exploratory semistructured interviews with local hospital network staff, quantitative project evaluation data and implementation outcome measure surveys will be used to test and refine CMOs, constructed in stage 1. This stage will follow a convergent (complementary qualitative and quantitative datasets) model, integrating and merging separately analysed data through side-by-side comparison and contrast.^51–54 The convergent model will include capturing narrative and experiential information from local hospital network staff to highlight the mechanisms of implementation in different contexts, and variations in outcomes from the value-based care initiatives. These will be tested against the hypothesised initial programme theories, by eliciting comments to confirm, deny and refine aspects of the theory. Concurrent to these qualitative approaches, we will examine the quantitative relationships between implementation processes and the relevant implementation and programme outcomes for different populations, circumstances and time points, according to the programme theories. Explicating these relationships between variables will build on the structure and process models created in stage 1, to articulate generative or conditional causal models.⁵⁵

Stage 3: translating CMOs into generalisable theoretical models for implementation (middle-range theories)

The process of testing programme theories in stage 2 will lead to the refinement of CMOs for the seven value-based care initiatives, which will be translated into generalisable theoretical models (middle-range theories) for implementing large-system, value-based care programmes in stage 3. The process of translation will begin by presenting and defending CMOs with key stakeholders. Further testing will be undertaken by explicitly seeking disconfirming or contradictory data and considering other interpretations that might account for the same findings. Cross-case comparisons will then be used to determine how the same mechanism could produce different outcomes in different contexts and settings. The research team will review our realist findings for each initiative (including programme theories and CMO configurations) ‘vertically’ to identify common thematic elements according to context, mechanism and outcomes. Data will also be analysed across case studies ‘horizontally’ to uncover potentially generative causal patterns or regularities between mechanism and outcomes under certain contextual conditions. This process will eventually translate the specifics of each individual initiative to more analytically generalised abstract theories.

Qualitative methods

Qualitative sampling and recruitment procedures

A purposive sample supported by snowballing will identify respondents best placed to cast light on the proposed programme theories. Interview invitation emails will be distributed from the investigators via the state-wide implementation support agencies and local hospital networks to identified staff involved in the implementation of initiatives. We intend to interview up to 30 implementers and evaluators from the programme implementation support agencies for stage 1, and up to 30 local hospital network staff in stage 2 (see table 4 outlining the data source for each stage). Realist studies rely less on the concept of data saturation because realist hypotheses are confirmed or abandoned using a multi-method strategy for relevance and rigour.^{31 56} Our iterative process of data collection will include the possibility of re-interviewing participants to further develop our programme theories as we become more knowledgeable about the programme through refinement against other data (documentary, interviews with other stakeholders and quantitative findings).^{34 48 57} Specific inclusion and exclusion criteria are provided in online supplemental Appendix 2.

Qualitative data collection

The realist interview builds knowledge of why there is variation around what happens in natural settings which helps develop, test and refine theories. The interviewer–interviewee relationship adopts the teacher–learner cycle beginning with the interviewer teaching ‘the particular programme theory under test’ to the interviewee and then ‘the respondent, having learnt the theory under test, is able to teach the interviewer about the components of a programme in a particularly informed way’.⁵⁸ Interviews will take place either face-to-face at the interviewee’s workplace or a mutually agreed location via telephone or videoconference, depending on the participants’ preferences. The interview guides will be pilot tested with two stakeholders. Interviews will be approximately 30–45 min in duration, undertaken by three experienced qualitative researchers (MNS, EFA, CP). Information from public and internal programme documents that is relevant to the implementation of the seven value-based care initiatives will be reviewed to inform the development of programme theories.

Qualitative data analysis

All audio recordings will be transcribed verbatim and entered into NVivo V.12 for data management.⁵⁹ Analysis will be undertaken by multiple researchers conducting the interviews, in a concurrent and continuous manner alongside data collection. Ongoing analysis work ensures early immersion in the data.³³ Consistent with realist approaches, data analysis will be retroductive in that it will oscillate between an inductive and deductive logic to multiple data sources, as well as incorporating the researchers’ own insights of generative causation for the programme theories.⁶⁰ After data collection, qualitative analysts will read through, and systematically analyse each transcript individually before engaging in group work consensus-building around CMOs and causal explanations.⁶¹ Qualitative analysts will then revisit our programme theories and refine these in the light of CMO data. Programme documents will be analysed alongside interview transcripts in NVivo to confirm or disconfirm our CMOs, ensuring a diversity of data sources and another level of clarification.

Quantitative methods

Programme evaluation data

Evaluation data relating to the implementation outcomes of value-based care initiatives and contextual factors impacting these processes will be accessed by researchers with institutional approval from the partner organisations. These data will be sourced from programme roadmap milestone measures, performance reporting systems, fidelity audits, hospital culture surveys, patient and carer surveys, and quality of life measures. Deidentified routinely collected administrative data related to programme implementation (eg, clinical processes indicative of implementation fidelity) will be collected from the Enterprise Data Warehouse for Analysis Reporting and Decisions (EDWARD), which is mandated business intelligence standard focused on reports for NSW Health Local Health Districts in running their operations. The specific outcomes of interest will be identified during stage 1 of the realist study pending both access and quality of available data.

Implementation outcome measure survey

The Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM) and Feasibility of Intervention Measure (FIM)⁶² will be administered via survey at a single time point across all local hospital networks. These measures include four simple questions at fifth grade readability level and do not require any specialist training to administer, score or interpret. Items are measured on a five-point Likert scale and scores are calculated as a mean. The AIM, IAM and FIM each demonstrate strong psychometric properties.⁶² Local hospital network staff involved in the implementation of initiatives will be invited to take part in the survey in an email distributed by the researchers, via the programme implementation support agencies. The survey will be administered online, using REDCap (Research Electronic Data Capture) tools hosted at Macquarie University.⁶³

Quantitative data analysis

Statistical analyses will be conducted using commonly available software packages to explore the impact of implementation strategies and processes on a range of implementation and programme outcomes which will be identified during stage 1 of the realist study depending on the access to and quality of data. Descriptive statistics will be used to present continuous data as mean and standard deviation (SD), median and (inter-quartile range (IQR), or sample maximum and minimum, and categorical data as frequencies and proportions. For the purposes of this realist study, programme implementation strategies and processes will generally be categorised as independent variables, where implementation and programme outcomes will be categorised as dependent variables. Interaction effects will be considered where appropriate.

Patient and public involvement

The views of Australian public representatives from the NSW Ministry of Health and implementation support agencies were incorporated into the design of the study, from the stage of research grant funding application. The research questions and observations of interest were co-developed in consultation with the project partners, informed by their experience implementing the initiatives of interest for this study protocol. A Project Advisory Group was established with representatives from local hospital networks to support the design and recruitment process for the study, minimise time burden for participation and identify pathways for disseminating findings across the health system.

Ethics and dissemination

Ethical approval has been granted by Macquarie University (Project ID 23816) and Hunter New England Human (Project ID 2020/ETH02186) Research Ethics Committees. All interview participants will receive written information and provide informed consent prior to data collection taking place (online supplemental Appendix 3). All interview participants will be assured of anonymity and confidentiality during all data management, handling and reporting processes. Ethical considerations regarding the use of routinely collected administrative data where it is impractical to obtain individual informed consent will be managed in accordance with the National Health and Medical Research Council (NHMRC) National Statement on Ethical Conduct in Human Research (2007)—Updated 2018.⁶⁴

Data storage and retention

Electronic data will be stored on a password-protected university server and accessible only to members of the study team. Hard copy data will be stored in locked cabinets within secure offices. Data will be destroyed seven years after the date of the final peer-reviewed publication, in accordance with ethical guidelines.

Dissemination

A series of roundtables will be conducted across Commonwealth, State and Territory health systems in Australia to extend the benefits from this programme of work. Special attention will be paid to hospitals, local hospital networks and States as health systems managers. The results from this project will be submitted for publication in peer-reviewed academic journals and will be presented at conferences and seminars with identifiable information omitted.

Discussion

This realist study will support the implementation of value-based healthcare improvement initiatives, by leveraging our understanding of complex adaptive systems,^{65 66} evidence-based medicine⁶⁷ and the emerging science of implementation.^{68 69} Successful systems-wide improvement depends on identifying, learning and capturing the benefits from local innovation and understanding that, in complex systems, local contexts are critical to success.^{24 70–73} Our investigation of programme implementation in varying populations, circumstances and time points will deliver a transferable model for implementing large-system improvements in healthcare. This evidence-based ‘road map’ of implementation practices is expected to enable local successes to be effectively translated into meaningful, systems-wide change.

Realist studies are based on the premise that it is not possible to create and measure entirely closed system experiments for healthcare research in real-world conditions. Protocols for controlled trials are usually produced to reduce the risk of outcome and reporting bias in the evaluation,^{74 75} and to enable replication.⁷⁶ However, it is not possible to describe every detail of a realist study in advance, as their iterative and exploratory nature mean that important factors for examination emerge as theory develops.⁴¹ Therefore, protocol deviation in realist studies does not necessarily constitute a risk of bias, given the inherent flexibility of realist study design. This realist study protocol will ensure rigour in planning, provide transparency of approach, support ethical considerations and demonstrate realist assumptions in our project conception.

An important consideration for our realist study is articulating the bounds of inquiry and finding a level of abstraction that allows the building of theories that are generalisable to other settings and implementation of similar programmes. This work focuses specifically on the implementation of seven LBVC initiatives, which have consistent overarching goals and a suite of implementation strategies. The proposed realist study will only provide insights towards the science and practice of implementation and will not necessarily articulate the overall benefit or value of the identified initiatives. A parallel investigation is being conducted by the project team to identify, measure and test an economic benefits measurement framework for value-based healthcare, using these initiatives, that articulates the value from expected health outcomes and preferences for experience and equity (Partington et al, forthcoming). It is intended that the realist study outputs will be at a level of abstraction that allows observations ‘across’ programmes to see where mechanisms are operating, and how different contexts for them lead to other outcomes, thereby supporting generalisation.