What do perioperative national clinical audits tell us? The evolving role of national audits in changing practice and improving outcomes

Learning objectives.

By reading this article you should be able to:

• •Describe the role and impact of national clinical audits in quality assurance and improvement.
• •Outline why it is essential to submit high-quality data to national audits for accurate benchmarking, and to underpin quality improvement and research.
• •Explain how data collected for audit may be used to understand local trends and measure the effects of change using quality improvement (QI) methods.

Key points.

• •National clinical audits (NCAs) seek answers to important questions about processes of care and outcomes in specific groups of patients.
• •National data are used to benchmark care and outcomes for quality assurance purposes.
• •Contemporaneous data and interactive tools support frontline clinicians to deliver quality improvement.
• •NCA data enable healthcare commissioners to prioritise funding.
• •The care and outcomes of some of the most complex and high-risk patients have been transformed over the life cycles of some of the most prominent NCAs.

Introduction

Perioperative national clinical audits and improvement programmes

Perioperative national clinical audits (NCAs) are funded and run in many countries worldwide by organisations that include the NHS and the Royal Colleges of Anaesthetists and Surgeons in the UK, corresponding bodies in the USA and Australasia, and the Swedish Rikshöft. Perioperative NCAs investigate and report on diverse services including maternity, critical care and surgery; and groups of patients such as those with chronic morbidity.

There is a variety of NCAs, reflecting their different objectives and subject of interest. Audits are most commonly started to provide quality assurance (QA) where inconsistencies in care, outcomes, or both are suspected or have been demonstrated. Many contemporary NCAs also incorporate quality improvement (QI) methodology, which may be defined as a process of iterative change using a the ‘Model for Improvement’.¹ The objective behind inclusion of QI initiatives is to increase value in healthcare systems, whilst ensuring the reliable delivery of high quality perioperative care (‘right care for the right individual at the right time, by the right person’) with improved outcomes and reduced costs.

Methods range from snapshot projects spanning a matter of days to embedded mandatory audits that run for many years; and from registries for the reporting of rare events to interactive multidisciplinary platforms that guide the routine provision of care to tens of thousands of patients every year.

In the UK there are currently more than 50 NHS-funded NCAs in progress. NHS funding bodies are the Healthcare Quality Improvement Partnership (HQIP) and NHS Digital. In addition, various colleges and professional bodies fund their own national audits. NCAs incorporating QI methodology are well established in the USA, most notably the far-reaching American College of Surgeons' National Surgical QI programme (NSQIP). NSQIP has extended its reach outside the USA to hospitals in New South Wales Australia, aiming to capture 75% of public sector surgical procedures statewide. NSQIP champions collaborative working and the use of data to drive improvement.

QI programmes are becoming embedded across the rest of Australia, New Zealand, and throughout Europe. Led by bodies including the Australasian Colleges of Anaesthetists and Surgeons, the interstate Australian commission on safety and quality in healthcare, and the Swedish Rikshöft (Swedish national hip fractures registry), projects such as the ‘NZ triple aim for quality’ continue to progress. In the UK, the perioperative quality improvement programme (PQIP) is a multidisciplinary initiative that uses both innovative QI solutions and QA benchmarking to benefit patients undergoing major surgery. In Denmark, the safer hospital programme was a national 4 yr project measuring change and improvement utilising twelve clinical ‘bundles’ (including surgical errors, sepsis and cardiac arrest). This programme sought to reduce overall hospital mortality and harm by the meaningful use of data in real time to identify gaps in quality of care.

There may also be opportunities to use QI methodology in developing health economies. In India, the government-funded National Health Mission, in conjunction with the National Health Systems Resource Centre, is leading implementation of a framework for defining standards of care and measuring and improving against these. There is particular focus on the public hospitals that may care for some of the most poor and vulnerable patients. The aim is to ensure these patients receive standards of care benchmarked against globally recognised standards. However, while it is well recognised that in low- and middle-income countries competition for scarce resources may be barriers to audit and improvement, improvement science can be used successfully with investment, collaboration and support from external sources and charities.² Beyond the scope of these national audit and QI initiatives, international collaboratives are developing, as with the second National Institute for Academic Anaesthesia (NIAA) snapshot national audit project.

Do NCAs improve care and outcomes for patients?

Improvements in outcomes and consistency of delivery of care have been observed alongside NCA programmes worldwide.3, 4, 5, 6, 7 Outputs of NCAs have contributed to fundamental changes in the way care is delivered in several ways: by developing new evidence bases that lead to advances in clinical practice; by comparing outcomes and challenging compliance with national standards that inform patients' choice (of treatment course, hospital and even lead clinician); by promoting the consistent delivery of key components of care by providing incentivises to the use of care bundles and pathways⁸; and by standardised high-quality care through the development of recommendations for standards of care.⁹ In addition to the QI methodology outlined, NCAs may use population health programmes, systems for feedback to clinicians, and observational research techniques in order to achieve their objectives.3, 7

However, it is difficult to isolate any direct impact of NCAs, requiring methods to control for the many potential interacting and confounding factors inherent in complex organic systems, and to distinguish improvements that might be associated with NCAs from general progressive improvements in care.¹⁰ The HQIP framework (Box 1) is an example of a tool designed to facilitate assessment of the impact of NCAs.

Box 1. The HQIP framework.

HQIPs NCA Impact Assessment tool covers four domains;

• 1.National (evidence of national improvements in the quality and outcomes of care)

Examples include: year-on-year national improvements in outcomes; reduction in deaths; improved care processes; or even decline or stagnation and how the project helps interpret it and/or makes recommendations.

• 2.Local (how the project stimulates quality improvement)

Examples include: improvement vignettes; action plans; improvement initiatives.

• 3.System (how the project supports policy development and management of the system)

Examples include: influence on national policy; relation to commissioning for quality and innovations (CQUINs); BPT; clinical commissioning group outcomes indicator set (CCG OIS); NHS outcomes framework (NHS OF); NICE; data sharing; data.gov.uk.

• 4.Public (how the project is used by the public and the demand for it)

Examples include: hits on website; press coverage; social media coverage; use by patient groups.

Alt-text: Box 1

The HQIP framework emphasises the identification and analysis of specific evidence of an audit having an impact at national, local, system and public levels. For example, evolution of national policy and recommendations (such as the provision of critical care beds) may be informed by NCA projects having demonstrated a shortfall in the number of patients admitted to ICU after high-risk surgery. Similarly, local system and staffing changes promoting early mobilisation of patients after a hip fracture have used the local benchmarking data provided by the national audit programme. Importantly, the data collected can be presented easily to executive teams and healthcare commissioners to improve care.

So, although it may be difficult to identify specific consequences of NCAs, the results of NCAs have been used to support paradigm shifts in service delivery through the dissemination of key messages, QA, and use of tools to facilitate local QI. Examples include transformations in care, reduced length of hospital stay and improved survival in some of the most complex and high-risk patients.3, 4, 5, 6, 7

In this article, we explore international examples of NCAs and associated initiatives (Box 2). We highlight common themes in their evolution, identify ways in which they have driven changes in clinical practice, and discuss associations with improved outcomes, with an emphasis on audits of perioperative care.

Box 2. Glossary of national initiatives and audit projects referred to in this article. Web addresses correct at May 6, 2019.

ACS	American College of Surgeons (https://www.facs.org/quality-programs)
DSPS	Danish Patient Safety Society (https://patientsikkerhed.dk/english/)
FFFAP	Falls and Fragility Fractures Audit Project incorporates national hip fracture dataset (NHFD), fracture liaison services (FLS) and national audit of inpatient falls (NAIF) (www.fffap.org.uk)
HQIP	Healthcare Quality Improvement Partnership (www.hqip.org.uk/a-z-of-nca)
IHI	Institute for Healthcare Improvement (www.ihi.org)
NELA	National Emergency Laparotomy Audit (www.nela.org.uk)
NIAA	National Institute for Academic Anaesthesia (www.niaa.org.uk)
NSQIP	National Surgical Quality Improvement Program (www.facs.org/quality-programs/acs-nsqip)
PQIP	Perioperative Quality Improvement Programme (www.pqip.org.uk)
Rikshöft	Swedish National Hip Fracture Registry (https://rikshoft.se/about-rikshoft)
SNAP	NIAA Sprint National Anaesthesia Projects (www.niaa-hsrc.org.uk/SNAPs; www.niaa-hsrc.org.uk/SNAP-2-EpiCCS#pt)

Alt-text: Box 2

Common themes

The groups of patients studied and methods used may vary substantially between audits, but there are many common themes in key findings, design considerations and efforts to drive improvement.

QA, QI, and research

The primary role of NCAs has traditionally been viewed as QA, assessing and reporting processes of care and outcomes as comparisons between hospitals and against standards of care. In the UK, NCAs do not set national standards but do publish recommendations aiming to guide hospital teams in how to provide care that matches recognised standards that are based on best practice and research. QA is important in demonstrating quality gaps in care provision and remains a useful tool in demonstrating shortfalls (and therefore need for investment) to commissioners and hospital executive teams.

However, using the traditional audit cycle, NCAs have been associated with only limited amounts of standardisation and improvement. So, in response to a recognised need to drive change and use the data in more timely and effective ways, many NCAs have moved away from producing lengthy annual reports towards providing contemporaneous, instantly available data to local teams alongside individualised benchmarking reports. This supports the parallel objectives of QI and research to expand the knowledge base and deliver effective, sustainable improvement.

Rigorous audit may be considered as being part of translational research.¹¹ The vast purpose-built datasets mean that NCAs are well placed to facilitate health services research, exploring the association between differences in organisational factors (processes and structures) and variations in outcome. Evidence suggesting the implication of individual organisational factors remains limited, but bundles and pathways of care that support consistent delivery of key processes of care may be associated with improved outcomes.8, 12 There are many examples of perioperative NCAs advancing the evidence base or opening up their datasets for researchers to do so.

Audit design

Without data you're just another person with an opinion

W. Edwards Deming

Data quality and data burden

Data quality underscores the credibility of the findings of the NCAs, and analyses are only as good as the data entered locally. NCAs have a responsibility to create refined datasets that are easy to use and fit for purpose. Incomplete documentation, leading to inaccurate coding, complicates retrospective capture of data.¹³ In addition to the obvious consequences for reimbursement for service, it detracts from understanding of current clinical practice and the ability to explore associations between processes and outcomes, as shown by the Falls and Fragility Fractures Audit Project (FFFAP).⁶ It also does not represent the patients' experience of their care.

It takes requires meticulous design and testing to create a dataset, with regular data sampling and measures to ensure completeness and accuracy. In seeking to define and measure practice, prospective audits could include vast numbers of data fields, but this creates a burden that risks compromising data quality and continued engagement. So where possible, NCAs use linkage with external datasets to achieve their objectives and reduce the data burden.¹⁴ The National Emergency Laparotomy Audit (NELA) and other UK NCAs use Office for National Statistics (ONS) mortality data and official hospital episode statistics (HES) admission and comorbidity data.

Using measures that matter

For NCAs to identify variation, stimulate change, and track progress, it is clear that processes, outcomes, and organisational structures must be carefully selected to report and compare. The ‘right’ measures are determined by the specific needs of each audit, balancing clinical relevance and QA objectives. Measures often also reflect ‘quality control’, if reported against targets or standards of care.

Outcomes are selected for relevance (to service users, clinicians, and commissioners); accuracy and transparency of measurement and reporting; and, in order to show variation, a sufficiently high incidence.¹⁵ Similarly, processes and organisational structures are selected for relevance (where evidence exists of association with variation in care and outcomes), accuracy of measurement and potential for QI (Fig. 1).

Fig 1.

National overall performance in hip fracture patients. Data extracted on December 13, 2018 from www.nhfd.co.uk. Reproduced with permission from FFFAP/Royal college of physicians (RCP).

Variation and consistency

Variations in outcomes (most commonly mortality rates) between hospitals and quality metrics are commonly reported by NCAs, and occasionally by the lay press. Variation also occurs within hospitals, both between teams (or lead clinicians) and over time. These data are increasingly included in hospital-level reports.

The identification of negative outliers in certain performance measures (such as mortality) is usually a contractual obligation of NCA programmes. They will formally notify the hospital teams and chief executive, and often have statutory responsibilities to inform regulatory bodies such as the Care Quality Commission in the UK. Increasingly, positive performing outliers are being identified. By subsequently interviewing both types of outliers, elements of best practice, key infrastructure, effective culture, or both at organisations with ‘good’ outcomes can be shared, whereas hospitals with ‘poor’ outcomes may be helped with measures to improve.

It should also be noted that each individuals' outcome is influenced by intrinsic risk factors. Hence when comparing institutions it is essential to adjust for case mix, to prevent hospitals treating a greater-than-average proportions of ‘high-risk’ patients being unfairly penalised. Funnel plots are a familiar way of reporting variation in outcomes; they account for chance variations and differences in case mix (Fig. 2).¹⁶

Fig 2.

Funnel plot of case mix adjusted 30 day mortality after emergency laparotomy—from the fourth NELA patient audit report (2018). Reproduced with permission from HQIP/NELA.

Reporting of variation over time is a central pillar of QI methodology, identifying whether changes in practice are associated with improvement.¹ Displaying variation, including using time series or control charts, enables healthcare professionals to readily assess and monitor process delivery, patient outcomes, or both through iterations of plan–do–study–act (PDSA) cycles (Fig. 3).¹⁷ Change may be more easily achieved and tracked at small scale, but QI methods may also be applied across regions or nationally.

Fig 3.

Example of simple run chart demonstrating changes in admission to critical care after operation from the NELA web tool.

Multisystem diseases and silos of knowledge

Every system is perfectly designed to get the result it gets

P. Batalden

As understanding of medicine has evolved, knowledge has become compartmentalised, with specialist teams working in silos. Complex patients with multisystem disease may be disadvantaged by these models of working but, in contrast, formalised shared care within multidisciplinary teams is associated with more consistent delivery of processes and improved outcomes. The FFFAP has demonstrated a progressive improve in quality and survival over the period of adoption of a collaborative orthopaedic–geriatric management of patients with hip fracture.

The prevalence of age-related conditions (including cognitive impairment and frailty syndromes) is increasing alongside the established increase in the incidence of multiple morbidities with advancing age.¹⁸ The NCAs have shown not only that older individuals (aged ≥65 or ≥70 yr) form the largest subgroup of high-risk surgical patients, but also that adverse outcomes are more common with advancing age and coexisting morbidity. Formalised input from physicians is known to be beneficial for orthopaedic patients undergoing surgery, and multidisciplinary working is well-established in diabetes, respiratory and stroke medicine.19, 20 However, coordinated multidisciplinary care is not routine for many high-risk surgical patients, including those undergoing emergency laparotomy. Our recent analysis suggested that this should be a focus for action, as multidisciplinary care pathways are associated with reduced variation in survival after emergency laparotomy.²¹ Perioperative medicine initiatives are gaining popularity worldwide, and these may contribute to modernising system design and function.²²

Predicting risk (and knowing what to do about it)

Adverse events may be predictable, whether intraoperative or postoperative, whether morbidity, mortality, or change in residential status. Identifying risk factors and quantifying their association with outcomes of interest using NCA datasets can inform the creation of contemporary risk prediction and risk adjustment models.²³ Examples in the UK include patients undergoing surgery for hip fracture, bowel cancer, and emergency laparotomy.

Accurate prediction models are used to identify high-risk patients, inform shared decision-making, and support proactive management before, during, and after surgery.

Setting standards and defining best practice

NCAs generate large volumes of patient data and rapidly accrue massive datasets. The topics studied by NCAs are guided by multidisciplinary expert panels of clinicians and based on current research, hot topics, and feedback from local clinical teams. Therefore NCAs are well placed to identify where best practice is being delivered. They can also track improvements, describe minimum standards of care and make recommendations for clinicians and healthcare commissioners. Although the evidence for the effectiveness of individual processes is conflicting at best, outcomes may be better when pathways or bundles of care are used routinely.¹² It is not known whether this relates to specific elements of the bundles, whether their uptake reflects differences in culture within an organisation, or their use is a marker of consistency of care.

Supporting and driving change

Many NCAs worldwide have led to improvements in outcomes, consistency of care, and funding for essential infrastructure.3, 4, 5, 6, 7 Nevertheless, while substantial initial improvements can occur, consistent investment and engagement is needed to effect permanent change.3, 24 Therefore NCAs have increasingly adopted QI principles and engaged with healthcare commissioners to gives incentives for the consistent delivery of best care.

Good support is need for organisations and clinicians to engage with and maximise the impact of QI initiatives. The QA aspect of NCAs can support this. By comparing data with nationally recognised standards, poor outcomes, potential shortfalls, and areas that require investment (time, personnel, and financial) can be demonstrated clearly to hospital executive teams and commissioners.

Drivers of change may be considered as being local, regional, and national, although there is substantial interplay between these. National reports can drive local changes and best practice at local and regional levels can drive national initiatives.

Local and regional

By facilitating QI, NCAs can empower clinical teams to combine expert system knowledge with local data to identify and test where change is required, evaluate the anticipated and unanticipated effects of interventions, and track progress of care and outcomes.²⁵

In order to support local clinicians to use their data for QI, contemporary NCAs operate bespoke data collection platforms that feedback readily usable data to clinicians to inform real-time clinical decision-making and support change, while fulfilling contractual QA roles. The NSQIP and NELA preoperative risk tools are examples of this. Dashboards that package feedback of local data as QI charts (including run charts and control charts) are used by audits including the FFFAP (Fig. 4), and many projects feedback raw data and provide tools and methods for participants to generate charts of their own local data. Some NCAs also allow for the addition of questions and themes to their core dataset to enable teams and networks to tailor and achieve their own improvement objectives. In the model used by NELA, these tools have been used to support both QI and research at a local or regional level.

Fig 4.

NELA data display available for clinical teams. The control chart uses control limits to monitor contemporaneous performance against a benchmarked metric.

QI should not occur in isolation. Knowledge, successes and best practice should be shared to facilitate widespread change and improvement. Collaborative projects and regional networks, using the Institute for Healthcare Improvement (IHI) model for collaboration, are being developed internationally including the academic health science networks (AHSNs) and NIAA quality, audit, and research coordinators (QUARCs) in the UK.²⁶

National

NCAs commonly have contractual obligations to provide benchmarking reports of hospital-level process and outcome metrics, which allow comparison between individual hospitals and against national ‘averages’. In addition to support for local and regional QI initiatives, national assurance data and evidence of engagement with NCAs can also be used by healthcare commissioning bodies to identify where investment in hospital infrastructure is needed most, whether staff skill-mix, inpatient capacity, or facilities.

The imposition of penalties for failure to achieve nationally set targets has achieved only limited improvement and reductions in variation of care. An alternative is to incentivise delivery of best practice bundles nationally. Practice may be measured and improvements tracked through the use of NCA data collection and reporting mechanisms. In the UK, the Department of Health best practice tariff (BPT) for patients with fractured neck of femur was associated with increased consistency of delivery of collaborative multidisciplinary perioperative care, using the FFFAP data collection platform.⁸ A BPT for emergency laparotomies was launched in April 2019, in collaboration with NELA (www.nela.org.uk/Best-Practice-Tariff).

What is the future for NCA programmes?

NCAs contribute to continued improvements in care and outcomes by reporting upon, making recommendations about, and raising awareness of a wide variety of conditions that may otherwise go unrecognised. However, if they are to continue to drive improvements, it is clear that NCAs must continue to evolve, using innovative solutions to maintain engagement with clinicians, facilitate real-time clinical decisions, and widen their scope to include patient-centred outcomes.

With the widespread adoption of QI science, participants and patients will expect data reporting to be available in near-real-time in order for NCAs to remain relevant and demonstrate value. Large annual reports that detail historical data will become outmoded, other than to identify temporal trends. Current NCAs are already adapting to support QI objectives by hosting QI tools and workshops to support data analysis and local reporting; presentation of complex data in intuitive formats; and in exploiting technological advances.

In conclusion, NCAs have contributed to changes in practice and improvement in outcomes worldwide. Beyond their traditional roles, modern projects support improvement initiatives and financial incentives to drive change.

Declarations of interest

Both authors are project team members for the National Emergency Laparotomy Audit. SH receives institutional funding for her role as the National Clinical Lead for NELA.

MCQs

The associated MCQs (to support CME/CPD activity) will be accessible at www.bjaed.org/cme/home by subscribers to BJA Education.

Biographies

Matt Oliver AFHEA FRCA PhD is a senior specialty trainee in anaesthesia and is the research lead for the UK national emergency laparotomy audit.

Sarah Hare FRCA is a consultant anaesthetist who is passionate about quality improvement and is the national clinical lead for the UK national emergency laparotomy audit.

Matrix codes: 1I05, 2A07, 3I00