Evaluating quality in clinical care

Abstract

Defining and maintaining quality is essential to surgical practice. It is only through structured approaches to assessing outcomes that we can ensure that optimal care is delivered. This article will define quality in healthcare and discuss assessment models with reference to pertinent surgical literature. National initiatives are discussed with a critical appraisal of their role and effectiveness. We discuss the aim of quality improvement initiatives and comment on reporting of outcomes. The difficult question of how to maintain quality during a crisis, such as an infectious disease pandemic, is addressed.

Introduction

Defining and maintaining quality is fundamental to surgical practice. The constantly evolving nature of the profession allows us to treat more complex pathologies with cutting edge therapeutic modalities. To ensure that the best care is provided to our patients, it is essential to define and evaluate quality within surgical practice.

High-profile scandals have caused considerable damage to the reputation of doctors and the National Health Service (NHS). The unmonitored practice of general practitioner Dr Harold Shipman, appalling systemic failures at Mid-Staffordshire NHS Foundation Trust and recent scandals in breast implants or gynaecological mesh repair have led to public mistrust in the quality of healthcare. The Francis report following the inquiry into Mid-Staffordshire NHS Foundation Trust called for a significant change to NHS culture, including organizational process to detect problems early, prevent patient harm, ensure accountability and promote a culture of continuous quality improvement.¹ The Bristol Royal Infirmary inquiry highlighted inadequacies at every point of the care pathway as well as a dysfunctional ‘club culture’ which resulted in excess death of children undergoing heart surgery. It is therefore imperative that quality of care is under appropriate scrutiny to ensure standards are maintained and patients achieve the best outcomes.

This article will discuss defining and monitoring quality in healthcare by focussing on the multifaceted approach required to ensure we appropriately scrutinize care delivery.

Defining quality

The World Health Organization defines quality in healthcare as ‘the extent to which health care services provided to individuals and patient populations improve desired health outcomes’.² Quality involves ensuring that health care is safe, effective, timely, efficient, equitable and people-centred.³ The Health Foundation suggest an approach which takes in to account these six dimensions when priority setting, so that excellence in healthcare can be achieved.⁴

In the NHS, there are multiple stakeholders who have a role in delivering healthcare quality, ranging from individual surgeons to trust boards, commissioners, the Department of Health, patients and the public. Defining principles outlined in the General Medical Council's Good Medical Practice require clinicians to provide accurate and up-to-date information about their clinical practice, which ensures that patient safety is held in highest regard.

Donabedian's model for measuring quality care is advocated by NHS Improvement and the Royal College of Surgeons of England (RCSEng) and described three main aspects of healthcare performance that can measure quality: outcomes, process and structure (Figure 1 ).⁵ Donabedian describes a linked pattern where structure measures impact on process measures, which in turn impact on outcome measures. The emphasis is on using these three areas in parallel as employing only one outcome measure will lead to inaccurate data capture.

Figure 1.

Three aspects of quality care.

Evaluating quality

Outcome measures

Outcomes measure the result of a healthcare process and can either be classed as clinical or patient outcomes. Clinical outcome measures are traditionally the most widely used as they provide quantitative synthesis of care quality in a format which can allow for direct comparison intra or inter speciality. Organizations can evaluate their outcomes through audit. The focus of audit is generally on the structure or process of care delivery. Standards are predetermined and are often well-recognized national or international outcomes, which should be achieved. Contributing to local or national audit allows an organization or individual to benchmark against agreed standards and highlights areas for improvement. The RCSEng advocate their members and fellows to continuously engage with national audit projects. The National Clinical Audit and Patient Outcomes Programme (NCAPOP) hosts centrally funded national clinical audit projects that collect data on compliance with evidence-based standards, and provide local trusts with benchmarked reports on the compliance and performance. This submission of data nationally allows for the heterogeneity across populations with the NHS to be represented and the analyses are more representative of true care delivery. A multi-centre study allows for greater power and increased validity of conclusions drawn.

The RCS Clinical Effectiveness Unit (CEU) was created in 1998 to provide a national centre of expertise in methods, organization and logistics of large-scale studies of the quality of surgical care. Since its inception, several major projects have been supported including the National Emergency Laparotomy Audit (NELA), National Prostate Cancer Audit and the National Oesophago-gastric Cancer Audit. The National Vascular Registry (NVR) embodies the success of collaboration amongst surgeons and the willingness to contribute data nationally to improve patient outcomes. The NVR is a national clinical audit which is commissioned by the Health Quality Improvement Partnership (HQIP) and run jointly by the Vascular Society or Great Britain and Ireland (VSGBI) and RCS CEU. Main index procedures within vascular surgery are included and data submitted by surgeons and interventional radiologists on process of care and outcomes. Trust level and surgeon-specific data on outcomes is available online in a transparent format providing patients with the ability to review them. The NVR Report was released annually and summarizes national outcomes highlighting areas of good performance and those areas that required improvement. Once suboptimal performance has been identified quality improvement can be initiated. The Getting It Right First Time (GIRFT) 2018 report identified significant pathway delays and resultant poor outcomes for patients with lower limb ischemia. Subsequently, the Peripheral Arterial Disease Quality Improvement Framework (PAD-QIF) was launched by VSGBI with the aim of improving revascularization and reducing amputation rates.⁶ This national framework is complemented by alignment of key parameters with the NVR leading to streamlined care for patients with peripheral arterial disease.

Performance indicators

Which outcomes should we use to evaluate a surgeon and a team's quality of care? How can we provide information on service quality for the profession and the public? Quality indicators such as perioperative mortality have always been considered key outcome measures; however, not all outcomes are universally applicable to all surgical procedures. The National Joint Registry (NJR) collates data on hip replacement surgery and this procedure has low associated perioperative mortality. The key outcome measure here is return to theatre and revision rates. Indeed, the NJR was instrumental in highlighting poorly performing metal on metal hip implants in 2008 and allowed for earlier medical device safety alert to be issued.⁷ Therefore, using multiple outcome metrics including 30-day readmission, unexpected return to theatre and length of hospital stay can provide valuable insight into how care is delivered. Expected performance of an indicator is based on research evidence, agreed standards of care and calculated from national registries.

Outcomes can be reported at organization level or at individual surgeon level. There was much controversy when surgeon level indicators were made publicly available in 2013 through an HQIP managed NHS England initiative called Consultant Outcomes Publication (COP). COP allowed patients the ability to review individual and trust level outcomes for key surgical procedures and aim to increase transparency in the NHS. Some suggested that a risk adverse culture would ensue to avoid negative statistics and ultimately patient care would be negatively impacted. Reporting outcomes at organization level would also effectively pool outcomes of all surgeons and perhaps dilute any negative outcomes of one particular provider. In vascular surgery, there has been no link between individual surgeon reporting and risk adverse outcomes for abdominal aortic aneurysm repair.⁸ However, surgeon-level reporting does not take in to account the structure of care within the institution. Technical skills are important but provision and quality of other services such as multidisciplinary team meetings, critical care services, speciality specific anaesthetic and nursing staff also directly impact outcome. Data precision is also in question when a surgeon performs low numbers of cases but this often not appreciated in outcome reports. Furthermore, in the era of more complex surgery dual consultant operating is more common and asks the question, to which consultant should the case therefore be attributed? The evidence for both sides of the argument remains disputed, but reporting both sets of outcomes is associated with quality improvement.

Involvement of public and patient groups in designing outcome measures is a well-accepted and valid approach. These patient-reported outcome measures (PROMS) can be validated and nationally available, such as for hip and knee replacement and inguinal hernia surgery. The key with this data is to analyse any change in PROMS pre and post procedure. Ensuring all procedures carried out positively impact quality of life will allow us to provide more cost effective and streamlined care. Similarly, positive patient experience has been shown to correlate with clinical effectiveness. Patient experience measures can be employed as a surrogate measure of patient-doctor communication and support improvement.

Case mix

When comparing different healthcare providers, a statistical adjustment is required to take into account variation in socio-demographic characteristics between patients and regions. Adjusting for case mix allows for fair comparison between healthcare providers and helps to ensure there is no discrimination against services who treat populations with more prevalent or advanced pathology. Unadjusted comparisons can lead to inappropriate investigation or inadequate reimbursement for outcomes-based payment systems. Some argue whether this methodological correction is appropriate, as services that have poor outcomes should be highlighted and consideration of increased investment made. However, this practice can help to reduce incidence of withholding intervention from patients who may be deemed as ‘hard to treat’ and ensure everyone is provided with healthcare irrespective of background and co-morbidity.⁹ Transparency in methodological calculation of case mix is also important. Detailing variables and covariables that can effect outcome which are used in case mix adjustment need to be agreed and validated otherwise they can lead to unfounded results.

Case mix and local quality priorities are also relevant when an organization is designing a quality strategy. Alongside national guidelines, a local strategic vision should be set out as this encourages continuous quality improvement.

Identifying and managing outliers

Each speciality has an ‘outlier policy’ which outlines robust processes for managing organizations or surgeons with indicator values that fall outside an expected range of performance. Associations such as the VSGBI have clear pathways and timelines set out for investigation and these processes should be transparent.¹⁰ Commonly, deviation from expected level of performance of between 2 and 3 standard deviations generates an ‘alert’ and more than three standard deviations, flagged as an ‘alarm’, indicates an outlier. Services which are negative outliers from national averages should hold internal audit reviews to evaluate the accuracy of data and to decide if it truly represents their practice and case mix. Once this has been established, a series of interventions can be implemented locally to improve quality.

The importance of local and national clinical governance was highlighted in 2018 when a novel endovascular aneurysm sealing device (Nellix EndoVascular Sealing System, Endologix, Irvine, US) resulted in a high incidence of mid-term failure rates.¹¹ Local audit was followed by rapid international dissemination of results which led to several other centres reviewing their outcomes. The Nellix device was then highlighted as having suboptimal performance and a safety notice was issued by the UK Medicines and Health Regulatory Authority (MHRA). Following this, the device was withdrawn from the UK market and lost its Conformité Européenne mark across Europe. This incident highlighted several key problems with medical device monitoring. Commonly, only 1-year safety outcomes are required for a device to become freely available for use. Clearly, the problems with this particular device were not evident until around three years after implantation. Perhaps monitoring of novel devices should happen in national registries, similar to the National joint Registry, and devices with poor performance could be highlighted earlier.¹² External initiatives such as Beyond Compliance are set up to assess the relative risk of a new medical product, such as a joint replacement, and ensure it has ongoing monitoring and assessment.

Process measures

Process measures are quality indicators derived from a specific process which has an influence on outcome. Occasionally, measuring outcome measures is difficult but process measures are generally easier to collate and provide insight on the actions by which outcomes are achieved. An example is the time from symptom onset to carotid endarterectomy, which has been shown to significantly influence the incidence of stroke and mortality.¹³ Process bundles which detail pathways such as the WHO checklist are also shown to improve surgical outcomes.

The NHS GIRFT initiative is one example of a national quality improvement programme which aims to improve quality of care and lower costs by reducing variation in services and practice. Other projects include Enhanced Recovery After Surgery (ERAS) and Day of Surgery Admission (DOSA). The National Emergency Laparotomy Audit concluded that both a consultant surgeon and a consultant anaesthetist should be in theatre when risk of death is =/> 5% and if risk is not documented then this is less likely to occur. The RCS Cholecystectomy Quality Improvement Collaborative (Chole-QuIC) was initiated following highlighting of significant variations in care in gallstone disease in emergency general surgery. This multicentre quality improvement project concluded in 2018 and resulted in streamlined treatments pathways timelines with more patients with acute gallstone disease undergoing cholecystectomy within 8 days of presentation.¹⁴ This led to considerable cost savings from reduced subsequent emergency admissions from gallstone disease and gallstone pancreatitis.

Structure measures

Structure measures, also known as input measures, reflect the attributes of a service. This includes organizational structure, clinical services and physical environment. In some specialities there is correlation between levels of activity and performance. It was suggested that centres with low volume of abdominal aortic aneurysm (AAA) surgery had poor outcomes when compared to high volume centres.¹⁵ Subsequently, the UK was found to have a high outlying mortality rate for AAA when compared with Europe (7.9% vs 3.5%) and this led to Abdominal Aortic Aneurysm Quality Improvement Programme (AAAQIP). The AAAQIP, in combination with GIRFT, recommended centralization of services so that AAA repair is only undertaken in centres that achieve minimum numbers. However, the RCS recognize that using purely case volume to prove quality is a ‘blunt tool’ and should not be used in isolation.

At regulatory level, the CQC is responsible for inspecting and regulating services to ensure that they meet fundamental standards of quality and safety. The CQC focus on organizations and not individuals and the inspection reports highlight both areas for improvement and those of excellence. It has been recognized that improvement does not simply occur because an area has been highlighted. The process is intricate and multifaceted; the focus is on equipping local services and systems with the skills to support success.

Other models for improving quality include economic drives such as the Commissioning Quality and Innovation (CQUIN) payment framework. A financial incentive is offered to organizations who meet standards such as accurate recording of smoking data in primary care or completion of venous thromboembolism risk scoring in secondary care.

Quality during a global crisis

In 2020, the world was confronted with the challenge of a global pandemic. COVID-19 posed significant challenges for healthcare services worldwide with large volume of admissions of critically ill patients. Maintaining quality during a time of crisis is an exceptionally challenging task.

To cope with increased demand, especially on critical care services, many clinicians have been redeployed to roles that are outside of their usual duties. Being provided with a formal induction to a new clinical environment is paramount and helps to provide orientation to new staff and signpost where to go for help. During unprecedented times, the GMC's Good Medical Practice should be remembered: a clinician should never do anything that is outside of their competence; even in exceptional circumstances. Several national organizations have made clinical materials available free of charge for those who are required to work in different specialities. Appropriate organizational processes are required, such as structures to support psychosocial wellbeing of staff during a time of immense psychological strain and stress. Furthermore, routine administrative tasks should be suspended to allow clinical teams to focus unwaveringly on their duties to patients.

An Intercollegiate Consensus was launched to provide surgeons with guidance on delivering care during COVID-19.¹⁶ The aim was to focus on acute patients and to balance risk of admission with that of nosocomial infection. Most centres suspended elective surgery to reduce hospital footfall and ease pressure on critical care services. However, provision for emergency operating has been maintained to an extent with appropriate personal protective equipment available. Common procedures have been modified, such as the reduced use of laparoscopic surgery and more open operations to reduce aerosol generation. National surgical specialities have issued recommendations on how to triage cases for operation. The VSGBI have advised that ruptured AAAs should be managed with EVAR and open surgery only considered if sufficient critical care capacity is available. Furthermore, timelines for urgent care have been published to aid surgical decision making. During the consent process surgeons need to introduce the impact of a pandemic on a patient journey and proposed treatment plan. Performing colorectal cancer resections with no plan for an anastomosis at the index procedure is an example of this which can alleviate the risk of leak and hence likely prolonged hospital stay and of nosocomial COVID-19 infection.

Planning of services after a pandemic is also essential. A global collaborative estimated over 28 million operations have been cancelled during a 12-week period of disruption due to COVID-19.¹⁷ It would take over 45 weeks to clear this backlog if surgical caseload volume was increased by 20%. It will take time to evaluate the impact of this pandemic on quality of care in surgical patients, but we must act on lessons learnt now to ensure we are better prepared in the future.

Conclusion

We strive to provide patients with the best quality of care, and in order to achieve this we must ensure that we engage with continuous quality assessment and improvement. Transparency is key for patients and clinicians alike and the challenge is to maintain this while the demand on healthcare services grows with an ageing population. Applying the multi-faceted approach proposed by Donabedian allows us to evaluate effectively the quality of service we are providing. Engaging with national audit and quality improvement initiatives and partaking in a culture of shared learning will give the opportunity to continue improve outcomes. Continued focus and research within this area will lead to more efficient and patient centred care.