A realist synthesis of multicentre comparative audit implementation: exploring what works and in which healthcare contexts

Abstract

Background

Multicentre comparative clinical audits have the potential to improve patient care, allow benchmarking and inform resource allocation. However, implementing effective and sustainable large-scale audit can be difficult within busy and resource-constrained contemporary healthcare settings. There are little data on what facilitates the successful implementation of multicentre audits. As healthcare environments are complex sociocultural organisational environments, implementing multicentre audits within them is likely to be highly context dependent.

Objective

We aimed to examine factors that were influential in the implementation process of multicentre comparative audits within healthcare contexts—what worked, why, how and for whom?

Methods

A realist review was conducted in accordance with the Realist and Meta-narrative Evidence Syntheses: Evolving Standards reporting standards. A preliminary programme theory informed two systematic literature searches of peer-reviewed and grey literature. The main context-mechanism-outcome (CMO) configurations underlying the implementation processes of multicentre audits were identified and formed a final programme theory.

Results

69 original articles were included in the realist synthesis. Four discrete CMO configurations were deduced from this synthesis, which together made up the final programme theory. These were: (1) generating trustworthy data; (2) encouraging audit participation; (3) ensuring audit sustainability; and (4) facilitating audit cycle completion.

Conclusions

This study elucidated contexts, mechanisms and outcomes influential to the implementation processes of multicentre or national comparative audits in healthcare. The relevance of these contextual factors and generative mechanisms were supported by established theories of behaviour and findings from previous empirical research. These findings highlight the importance of balancing reliability with pragmatism within complex adaptive systems, generating and protecting human capital, ensuring fair and credible leadership and prioritising change facilitation.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• Multicentre comparative clinical audits have the potential to improve patient care but can be challenging to conduct within resource-limited, heterogeneous healthcare environments. There are little data on what facilitates the successful implementation of multicentre audits.

WHAT THIS STUDY ADDS

• This study identified four main factors as influential in the implementation process of multicentre comparative audits within healthcare contexts: (1) generating trustworthy data; (2) encouraging audit participation; (3) ensuring audit sustainability; and (4) facilitating audit cycle completion.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• This study describes the contexts, mechanisms and outcomes influential to the implementation processes of multicentre comparative audits in healthcare environments. These findings will be useful to those planning these large-scale interventions.

Introduction

Multicentre comparative clinical audits involve the collection of data from several individual contributing centres which provide patient care for a given healthcare condition. Analysis of these data provides an overall picture of care standards on a regional, national or international level. This facilitates benchmarking, allows an informed approach to resource allocation and service design and has the potential to improve clinical outcomes and equity of care.^{1 2} Several multicentre audits are currently in operation internationally. Within the UK, the UK National Clinical Audit and Patient Outcomes Programme, commissioned by the National Health Service (NHS), manages 30 national audits which monitor important health conditions.

National audits have been associated with improvements in clinical outcomes. In South Africa, annual participation of 40 diabetes services in a multicentre clinical audit led to improvements across seven of nine key performance indicators for diabetes care.³ A study reporting on a similar French national prospective audit of diabetes care found an improvement in mean HbA1c and adherence to recommended screening a year after implementation of a full audit cycle.⁴ A report on the Scottish Hip Fracture Audit—an audit that was discontinued 6 years after introduction due to funding issues and reinstated shortly afterwards—found a steady improvement in key parameters relating to hip fracture care in the 6 years following audit implementation, a plateau after audit discontinuation and an upward improvement trend on audit reintroduction.⁵ Although these studies were not designed to establish causality, it appears that multicentre audits are at least associated with improvements in clinical care. They can also detect important interservice variations in provided care. For example, the UK National Diabetic Foot Audit, the UK National Pregnancy in Diabetes Audit and the UK National Lung Cancer Audit revealed unacceptable national variation between services in patient care.^6–8 In all cases, the audit committees disseminated recommendations that highlighted the need for service development in the underperforming regions. In addition, multicentre audits identify high-performing outliers: an interrogation of data from the UK National Diabetes Audit elucidated factors associated with high-quality diabetes services, such as insulin pump use and blood glucose strip prescribing. While these findings are again associations, they can guide further research that could lead to improvements in care.⁹

Multicentre audits are therefore likely equipped to identify areas of suboptimal care, to spark investigations behind discrepancies, to identify factors associated with high-quality care and to redirect funding appropriately. However, conducting audit in any form is not easy. Several studies explore the challenges of conducting audit at local level: facilitators include electronic medical records and open dialogue between providers and consumers; barriers include poor communication between stakeholders and lack of clear governance structures.^{10 11} The influence of these factors may be magnified when performing collaborative audit on a large scale. The number of existing national clinical audits suggests that implementing large-scale audit is feasible. However, there are a number of multicentre or national clinical audits which were not as successfully implemented.^{5 12–15} There are little published data to explain how and why some multicentre clinical audits are successfully implemented and sustained while others are not. Healthcare interventions are implemented within social systems affected by contextual factors that include individuals, interpersonal relationships, institutional settings and organisational infrastructure. Implementing a complex intervention such as a multicentre comparative audit is therefore likely to be a particularly context-dependent process.

Realist philosophy suggests that social realities and the interventions applied within them are complex and change depending on context. How an intervention unfolds in a given situation depends on generative mechanisms: ‘underlying entities, processes, or social structures which operate on particular contexts to generate outcomes of interest’.¹⁶ Realist synthesis (also called realist review) is based on this philosophy and aims to extrapolate from empirical studies the contextual factors (C) that trigger the mechanisms (M) that generate an outcome (O) when an intervention takes place within a given context(s).¹⁷ Mechanisms may be more likely to produce positive outcomes in certain social, organisational or cultural environments than others.^{18 19} These ‘context-mechanism-outcome’ (CMO) configurations are the essential outputs of realist synthesis—how, why, for whom and when an intervention is successful or unsuccessful.²⁰ As described above, implementing a multicentre comparative clinical audit is inherently challenging as it involves rolling out a large-scale project within multiple heterogeneous contexts. Realist synthesis has been frequently used in health policy research to help understand complicated and context-dependent interventions such as this.^{16 21}

Methods

Realist review has five steps which were followed in this study.²² The process is iterative, with new data used to test and refine emerging theories.¹⁶ All processes were conducted in accordance with Realist and Meta-narrative Evidence Synthesis quality standards for publication of realist reviews.¹⁶

Developing a preliminary programme theory

Programme theories are explanations for why an intervention might or might not work in certain contexts; in realist review, CMO configurations are the analytical blocks used to build these theories^{19 23} (table 1). After clarifying the scope of the review, a literature search facilitated the development of an initial programme theory around the potential contextual factors influencing the implementation of multicentre comparative audits. This literature search was based on the prior knowledge of the research team and a preliminary PubMed search. The search was not exhaustive but provided an overview of potential important factors. A further scoping search grouped emerging theories into a series of ‘if-then’ statements, which were refined to produce a final preliminary programme theory.²⁴ Regular discussions within the research team facilitated this process.

Table 1.

Definitions of context, mechanism and outcome

Category	Definition	Rule for coding
Intervention	The large-scale complex intervention implemented, being implemented or intended for implementation into a social or cultural context.	Intervention was considered as any audit discussed in a paper that had ‘passed’ the selection process.
Context	Any conditions relating to the social or cultural context in which the intervention is implemented that are likely to have an influence on mechanisms (below).	All components of context were considered: social, physical, interpersonal, structural, etc.
Mechanism	Causal processes triggered in a particular set of conditions that have an influence on the outcome (below) of a certain intervention.	Mechanisms were considered as any change in behaviour/process/decision-making/attitudes triggered by a particular contextual process.
Outcome	The outcome of implementing a certain intervention within a particular context, as governed by the above generative mechanisms.	Outcomes were considered as ‘end results’ of the audit implementation process.

Searching for evidence

Search criteria were designed to identify heterogeneous primary evidence sources relevant to the programme theory, that is, articles that focused on the ‘implementability’ of national or multicentre comparative audits within a healthcare context (online supplemental information). Using Boolean operators ‘or’ and ‘and’ to balance specificity with sensitivity, subject heading and keyword search strings were developed to reflect our preliminary programme theory and research question. The first search had two discrete search strings; one with an emphasis on potential contextual factors, the other on potential outcomes. The search strings were constructed for the MEDLINE database (PubMed) and then adapted to five additional databases: EMBASE, CINAHL, ABI-Inform, Web of Science and Cochrane Libraries. Emerging elements of theory warranted a second full literature search, with search strings constructed to explore these findings further. Searches were limited to English language articles with no publication date restrictions. Hand searching of reference lists was also performed. Grey literature was also searched, with a focus on relevant audit, clinical and government organisations.

Results from the two searches were transferred to the reference manager software SciWheel and deduplicated. The abstracts and titles of articles were screened using inclusion and exclusion criteria that were developed to ensure that selected articles focused on the audit implementation process (table 2). Abstracts that passed the first screening process underwent full-text review using a second set of selection criteria. These criteria included an appraisal of article credibility and its relevance to the emerging programme theory as judged by its relevance, rigour and trustworthiness (table 2). Articles were deemed relevant if it was felt that they could contribute towards the development of the final programme theory. Data were considered trustworthy if they were transparently and empirically obtained. An assessment of rigour considered the credibility of methods used to obtain data. For both searches, the primary researcher (SM-B) reviewed all articles and a second reviewer (SB) received a random sample of 10%. Disagreements were resolved by discussion. A flow diagram of this search and appraisal process is shown in figure 1.

Table 2.

Study characteristics

Study characteristics	Number of studies (n=69)
Country
UK	31
Australia/New Zealand	12
USA	11
Canada	8
Ireland	4
Other	3
Study type (n)
Cross-sectional	17
Cohort	16
Editorial/discussion paper	15
Other	11
Qualitative	10
Health problem
Other	35
General practice	12
Diabetes	7
Cardiovascular	6
Surgery	5
Mental health	4
Level of care
Secondary/tertiary	42
Primary	16
Mixed	11

Figure 1.

Search strategy—flow diagram: the first search, conducted in two discrete parts, reflected elements of the preliminary programme theory. For feasibility and efficiency, it was conducted in two discrete parts. 57 full-text studies were included for analysis. The first search revealed audit feedback to be a pivotal factor in audit implementation. The second search incorporated feedback into the search strategy.

Extracting data

A data code manual was developed to formally define the concepts of context, mechanism and outcome^{25 26} (table 2). A data extraction tool was designed based on existing recommendations and piloted on 10 records²⁷ (online supplemental information). For each article selected for inclusion, data were extracted by the primary researcher (SM-B) and a random selection of 10% was checked by a second reviewer (SB). Data with the potential to clarify, refine or refute emerging CMO configurations were selected.¹⁹ Relevant sections of text were identified as relating to context, mechanism or outcome.

Data analysis and synthesis of a refined programme theory

Data analysis was conducted using NVivo (V.12.1, QRS International) software. Framework analysis was used to qualitatively analyse extracted data (online supplemental information).²⁸ Data extraction forms from the 69 selected articles were read twice to facilitate data familiarisation.²⁹ An initial thematic framework was deduced from the preliminary programme theory to create a priori codes or ‘nodes’. Indexing of data followed, where sections of text from data extraction forms were coded against these a priori nodes and into overarching categories of context, mechanism or outcome. Several nodes were also inductively derived during this first round of coding, reflecting the fact that the initial thematic frameworks of framework analysis are tentative and need to be refined during the data analysis process.²⁸ Coding was an iterative process, with previously analysed articles regularly revisited and coded against newer nodes.²⁰ Charting of indexed data followed which involved further rounds of hierarchical coding against increasingly refined categories and subcategories.

Mapping data

Using contextual factors as index points, relationships between contexts, mechanisms and outcomes were identified to yield final CMO configurations.

Results

The preliminary programme theory

Early iterations of the preliminary programme theory were refined into a series of ‘if-then’ statements which were structured into an overarching preliminary programme theory (table 3).

Table 3.

Preliminary programme theory ‘if-then’ statements

Title	If	Then	Outcome
Navigating varying Information and Communications Technology (ICT) systems across hospitals (electronic, paper based, etc)	If you have clear and transparent processes for data collection, analysis and feedback.	Then the data yielded by the audit are trustworthy and reproducible.	And the audit can reliably guide service redesign and resource allocation to reduce geographical variation in care.
Audit implementation across several geographically disparate hospitals	If you have clear leadership at local and national levels.	Then all stakeholders across the country have the opportunity to meaningfully participate in audit design and implementation.	Which will result in sustainable buy-in and engagement of stakeholders with the audit process.
Audit implementation in a resource-limited healthcare environment	If you ensure strategic financial planning from the outset.	Then there is potential to direct funding to identified areas in need of resources or service development.	Which results in an audit that can drive quality and improvement nationally.
Disenchanted and disengaged staff	If you incentivise and collaborate with stakeholders at all levels from the outset of audit design.	Then stakeholders will feel valued and empowered.	Which will result in sustainable buy-in and engagement of stakeholders with the audit process.
Overworked staff in busy clinical environments	If you establish clear guidelines and operational support for those responsible for data collection.	Then staff will not feel overwhelmed by the audit process, or see it as a ‘tick-box’ exercise.	Which will result in sustainable buy-in and engagement of stakeholders with the audit process.

Search outputs

The first search reflected elements of the preliminary programme theory. For feasibility and efficiency, it was conducted in two discrete parts. 57 full-text studies were included for analysis. The first search revealed audit feedback to be a pivotal factor in audit implementation. Our preliminary programme theory and first search strategy did not emphasise feedback and our second search was conducted in response to this. This second search yielded 48 articles; many were duplicates from the first search, but 12 new full-text papers were added to the analysis. For both searches, the primary reviewer (SM-B) reviewed all articles, and a second reviewer (SB) reviewed a random sample of 10%, with an agreement rate of 95% (search 1) and 97% (search 2). The flow diagram summarises the search outputs (figure 1).

Study characteristics

There were 69 studies included in the final realist synthesis. Study characteristics are outlined in table 1.

Data analysis

Indexing of extracted data resulted in a large-volume codebook of 329 nodes and subnodes. Charting of indexed data involved second, third and fourth rounds of hierarchical coding to produce final themes (online supplemental information). The CMO relationships derived during the mapping stage were restructured using ‘outcome’ thematic categories as index points and working backwards. The final programme theory was generated using resultant CMO configurations, discussed in table 4 and figure 2.

Table 4.

Context-mechanism-outcome configurations

Context	Mechanism	Outcome
Defined core data processes	Clarity	Trustworthy data
Clear governance	Building capability
Operational support	Accountability
Quality-assured data	Reassurance
Secure data
Operational support	Acceptability	Stakeholder participation
Clear governance	Perceived legitimacy
Resource-limited contexts	Motivation
Piloting	Locus of ownership
Heterogeneous contexts	Patient empowerment
Multidisciplinary collaboration	Perceived fairness
Distributed leadership
Defined core data processes
Transparent data
Operational support	Supportive environment	Audit sustainability
Incorporating existing structures	Responsiveness
Operational support	Integration
Piloting	Adaptability
Financial support
Effective feedback	Accountability	Audit cycle completion
Action planning	Motivation
	Perceived fairness
	Building self-efficacy
	Responsiveness
Efficient process	Driving audit sustainability	Improvement in clinical care
Empowered stakeholders	Encouraging audit participation
Engaged stakeholders	Ensuring trustworthy data
Participant trust	Facilitating audit cycle completion

Figure 2.

The four context-mechanism-outcome (CMO) configurations. Yellow, context; blue, mechanism; green, outcome.

CMO 1: ensuring trustworthy data

Reliable data governance structures and data security measures contributed towards trustworthy audit data. Often, data coordinators were charged at local level with a responsibility for overseeing the collection of reliable data, facilitated by mandatory training.^{2 30 31} Several successfully implemented audits emphasised data security. Some audit processes relied on encrypted data collection tools to enhance security.^{2 15 32} Others conferred an external data processing group with the responsibility for ensuring data security.³³ A further mechanism was the use of pseudonymised data when comparing results between centres.^{30 34 35} These measures gave rise to a visible accountability for the safety of audit data.

In many contexts, quality assurance processes facilitated the collection of trustworthy data. Some were third-party verification processes, where data were sent externally for quality review.^{2 33} Other studies described internal quality assurance procedures, such as two-person data entry systems where data were reviewed or ‘double-checked’ by a second individual to ensure reliability.^{5 34–36} Another approach was the internal review of unexpected data prior to central submission, conducted manually by clinical staff^{30 31} or semiautomatically using audit filters that flagged outlying results.^37–39 These processes generated a visible accountability for the quality of audit data.^{39 40} Implementation processes that did not incorporate quality assurance structures reported a reduction in the actual and perceived trustworthiness of their audit data.^{41 42} This loss of trust limited the use of data in service planning decisions.^{43 44}

Defined data collection processes which were easy to follow facilitated data accuracy, completeness and trustworthiness.30 33 43 45 46 Vague definitions impeded data collection.^{37 47} In these instances, participants became frustrated by the data collection task and entered incomplete or erroneous data.12 40 43 48 Ongoing operational support facilitated problem-solving and troubleshooting of issues encountered during data collection.30 33 49–52 Mostly, this existed as virtual communication between audit participants and a clinical audit office via teleconference monthly,⁵³ weekly³⁸ or as needed.^{30 33 54} Clinical audit offices often were the source of operational support, but an open communication conduit between the audit office and participants was not a given; one study cited a lack of this communication as a barrier to effective operational support.⁴³

CMO 2: encouraging audit participation

Stakeholder motivation was a key driver of audit participation. Mandated governance procedures extrinsically motivated stakeholders to engage with the audit process. Mandatory participation was a feature of several audits—including 29 of the 70 national audits operating in the UK.2 43 49 55 Several audit processes, including the existing UK National Paediatric Diabetes Audit, incorporated outcomes-based commissioning based on audit results which encouraged participation.⁵² Multidisciplinary collaboration and distributed leadership were sources of intrinsic motivation and facilitated a sense of collective ownership driving audit participation. Indeed, if an audit was perceived as being ‘imposed’, resistance to the process rose over time.¹¹ Shared, fluid and collective leadership was an antidote to this.^{49 56–58} Local leaders were seen as ‘change champions’.⁵⁹ These leadership roles were either assigned or filled by individuals naturally engaged in the audit process.³⁰ By explaining to stakeholders how the audit would facilitate their roles at a local level, a sense of a shared organisational vision and collective ownership of the audit process developed.^{49 60} Local leadership required complementary higher level leadership to sustainably facilitate audit participation as it enhanced the perceived legitimacy of the audit process and assured local participants that collective audit ownership extended to a national level.^{30 41}

Multidisciplinary collaboration during audit design via workshops, focus groups, Delphi processes and audit committee meetings was a prominent feature of many successful audits, enhancing stakeholder motivation,⁶¹ increased trust in the process,⁶² perceived relevance⁴³ and autonomy.⁶³ Maintaining stakeholder collaboration throughout the audit implementation process cultivated the sense of audit collective ownership.⁵¹ This was achieved by regularly scheduled workshops to review data collection processes,⁵⁴ feedback strategies³⁵ and general user experience of audit participation.^{64 65} ‘Tick-box’ involvement of staff and service users, or lack of their involvement at all, resulted in poorer quality data and lower commitment to audit participation.⁶⁶

The majority of studies reported some degree of resource limitation within the audit implementation environment—a lack of funding,⁶⁷ time⁴³ or both.⁶⁸ If insufficient resources were provided to enable operation support for the audit, the increased workload was perceived as unfair and stakeholders disengaged from the process.⁶⁹ Efforts to redirect resources to ease the audit burden were viewed favourably by stakeholders and facilitated participation.³⁰ Data collection processes that acknowledge and alleviate resource constraints increase stakeholder acceptability of the audit workload.1 15 55 70

CMO 3: driving audit sustainability

Audits that incorporate existing organisational structures into audit design are more likely to result in a sustainable audit. This was achieved by making use of ‘off-the-shelf’ audit tools that had worked in similar contexts,³⁰ exploiting data routinely collected already^{31 50} and availing of existing communication conduits to connect with stakeholders.^{55 59 71} Piloting enabled existing structures to be identified prior to audit implementation,30 34 50 72 and was particularly effective if it resulted in iterative refinement of the audit process until it was appropriately tailored to the intended implementation environment.2 64 66 69

The availability of ongoing operational and financial support for audit processes was also important for audit sustainability. Ring-fenced, reliable funding streams were needed if an audit was to weather several audit cycles.2 30 33 59 This facilitated the provision of meaningful ongoing operational support from clinical audit offices, both virtual⁷³ and face to face.³⁰ The absence of ongoing operational and financial support usually resulted in disintegration of core audit processes and ultimately audit discontinuation.5 14 15 61

CMO 4: facilitating audit cycle completion

Feedback of audit data was broadly seen as a key driver of change, but opinions on what constituted effective feedback varied. Some studies identified transparent and public feedback processes as key facilitators of quality improvement; in these studies, audit data were fully transparent, identifiable and publicly available.^{39 40 74} Other approaches prioritised perceived fairness. Some audit feedback processes involved pseudonymisation of contributing hospitals, which facilitated benchmarking but prevented identification of individual centres.^{35 41} Others published audit data alongside service-level data describing the resources allocated to each participating centre, allowing for case mix adjustment and a fairer picture of clinical performance.^{33 38 73} An alternative approach aimed to reduce ‘competition’ between centres, delivered feedback privately to contributing centres, relying on local motivating processes to drive change.43 54 60 75 76 These different feedback processes facilitated improvement in their respective contexts, precluding selection of a single method ‘best’ equipped to facilitate audit cycle completion. Broadly, feedback processes that were overly critical created a blame culture that resulted in disenchanted participants and an audit that was not capable of generating sustainable quality improvement.^{11 59 77} Formative feedback that was non-punitive, helpful and task specific facilitated the intrinsic motivation of stakeholders to improve care quality.38 45 78 79

It was also key to incorporate action planning into feedback processes.^{59 74} Action planning involved providing participants with recommendations for action at the time of audit feedback.45 48 53 59 This facilitated change by increasing participant self-efficacy⁷³ and by empowering participants to problem-solve solutions to identified issues.⁴⁰ Action planning was particularly effective if conducted face to face within outreach workshops facilitated by those centrally overseeing the audit.^{51 59} Passive dissemination of feedback often resulted in limited actionability of audit findings.12 14 68 80 81

Discussion

This realist synthesis identified the CMO configurations associated with multicentre comparative audit implementation to be composed of several interdependent, interlinked and varied factors. Taken together, these CMO configurations highlight several important points, discussed below.

Balancing reliability with pragmatism within complex adaptive systems

Several elements of the final programme favour a pragmatic audit design, including adaptability, integration, making use of pre-existing structures, piloting and flexible data collection processes. Taken together, these factors suggest that a ‘one size fits all’ approach does not work well when implementing a multicentre audit in any healthcare context. Healthcare environments are examples of complex adaptive systems, in that they are composed of diverse, frequently shifting individual elements that interact autonomously and non-linearly.⁸² Singularly designed, top-down delivered interventions do not work in complex adaptive systems; rather, such systems call for iteratively designed implementation approaches that adapt to and evolve with particular contexts.^83–85 Accordingly, if a large-scale intervention such as a multicentre audit is implemented across several healthcare contexts, this will result in a product that differs slightly in its execution from one centre to another.^{83 86} This is not a flaw, but a feature of implementation within complex adaptive systems.⁸⁷ The challenge is to facilitate this adaptability while maintaining the core processes that define the essence of the intervention. This concept was reflected by our study finding that preserving a core dataset and defined governance processes were required for audit efficiency and reliability.

Generating and protecting human capital

An overarching finding was the importance of human capital to audit implementation processes. Many components of effective audit design were described, such as data security processes, operational support and feedback. However, these were ineffective in isolation and inextricably linked with a requirement for ongoing engagement and empowerment of audit participants. Processes that generated accountability were a source of extrinsic motivation. However, sources of intrinsic motivation such as collaboration, formative feedback and action planning emerged as even more important. Redolent of cognitive evaluation theory, this suggests that perceived autonomy is pivotal for the maintenance of motivation.^{88 89} Importantly, the maintenance of motivation is not a given; human capital, once activated, needs to be protected. Human behaviour tends to favour intrinsic motivation but requires a nurturing environment to do so.⁸⁸ Contextual factors that facilitated this in our study included the provision of adequate resources and operational support (for audit participation) and action planning (for audit cycle completion).

Leadership: getting the balance right

Rolling out a multicentre, comparative clinical audit requires the coordination of several implementation efforts across heterogeneous contexts. We found that leadership helps coordinate these efforts. Distributed leadership, with change champions at local level and senior clinical staff in national leadership positions, was a key contextual factor emerging from the realist synthesis that encouraged audit participation: new innovations require innovators (senior clinical staff pioneering the process at national level) but also early adopters (local change champions) to spread the message across a social system.⁹⁰ The need for local as well as national leadership in the audit implementation process is unsurprising if we consider that we are more influenced by those with whom we share a perceived professional or social similarity.⁹¹

The importance of effective feedback to audit cycle completion

Feedback is an integral part of the audit cycle and was almost universally acknowledged as a vital facilitator of clinical change within cases examined for the realist synthesis. However, as discussed earlier, opinions on what constitutes effective feedback processes were divided, particularly when it came to the degree of transparency of audit results. This variation likely stemmed from differing views of where the locus of accountability in healthcare should lie.⁹² Some audit processes favoured externally enforced accountability, where results were openly published and available for scrutiny. Others provided feedback privately to contributing centres on an individual basis, relying on sources of internal accountability such as professionalism and intrinsic motivation. Both approaches led to positive change in differing contexts, echoing previous studies that concluded feedback processes to be complex, socially constructed phenomena with effects highly dependent on their implementation environment. However, overindividualised feedback, particularly if summative rather than formative, cultivated a blame culture that acted as a barrier to quality improvement and audit cycle completion.^93–95 This reflects findings from other empirical studies and highlights the importance of emphasising collective, institutional accountability rather than personal accountability.^{92 96–98} While the approach to and efficacy of feedback varied, the requirement for facilitated action planning accompanying it in order to generate positive clinical change was a reproducible finding.

Strengths and limitations

Publication bias may have influenced the outputs of the realist review search as it is possible that cases of unsuccessful audit implementation are less likely to be published, meaning that important barriers to implementation may have been overlooked.⁹⁹ This was compounded by the finding that reports on existing clinical audits most often focused on results of the audit rather than factors influencing its implementation. Despite efforts to preserve methodological rigour, the interpretative nature of analysis used introduced an element of subjectivity and posed a risk of researcher bias.^{100 101} Furthermore, most of the literature selection and appraisal was conducted by a single researcher (SM-B), leading to potential bias, although efforts were made to reduce this using a two-step reviewer process for a proportion of all studies included. A strength of this study was its methodological design and choice of realist review approach, centred around an understanding that consideration of organisational context would be essential to effectively plan for the successful implementation of any multicentre comparative audit.

Conclusion

This study elucidated contexts, mechanisms and outcomes influential to the implementation processes of multicentre or national comparative audits in healthcare. The relevance of these contextual factors and generative mechanisms were supported by established theories of behaviour and findings from previous empirical research. These findings highlight the importance of balancing reliability with pragmatism within complex adaptive systems, generating and protecting human capital, ensuring fair and credible leadership and prioritising change facilitation. While the conceptual framework offered by this study requires validation in future research studies, data from this realist review have already been incorporated into real-life feasibility studies (eg, the recent National Paediatric Diabetes Audit Feasibility Study Report within Ireland¹⁰²). Going forward, it will continue to provide guidance for the development and implementation of multicentre comparative audits within heterogeneous healthcare contexts.

Data availability statement

All data relevant to the study are included in the article or uploaded as supplementary information.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

Ethical approval was obtained from the Trinity College Dublin Health Policy and Management Centre for Global Health Research Ethics Committee. Ethical approval was also obtained locally from the Children’s Health Ireland at Temple Street’s Research Ethics Committee.