Audit: how to do it in practice

Abstract

Junior doctors can find the process of doing an audit helpful in gaining an understanding of the healthcare process—here’s how to do one

Key points

• Audit measures practice against performance

• The audit cycle involves five stages: preparing for audit; selecting criteria; measuring performance level; making improvements; sustaining improvements

• Choose audit topics based on high risk, high volume, or high cost problems, or on national clinical audits, national service frameworks, or guidelines from the National Institute for Health and Clinical Excellence

• Derive standards from good quality guidelines

• Use action plans to overcome the local barriers to change and identify those responsible for service improvement

• Repeat the audit to find out whether improvements in care have been implemented after the first audit

In the United Kingdom, doctors in the first two years after graduation are asked to perform an audit. Audit measures practice against standards. Unlike research (which asks the question, “what is the right thing to do?”), clinical audit asks, “are we doing the right thing in the right way?”¹

Clinical audit forms part of clinical governance, which aims to ensure that patients receive the best quality of care. Clinical governance is often defined as how NHS organisations are accountable for continually improving the quality of their services and safeguarding high standards of care.²

Audit can include assessment of:

• The structure of care—for example, resources such as the presence of a dedicated stroke unit

• The process of care—for example, waiting times in clinics

• The outcome of care—for example, blood pressure reduction in response to therapy.

Audit should also be transparent. It should not be confrontational or judgmental—it is not an opportunity to name, shame, and blame.

Does audit work?

There is conflicting evidence on whether audit works. Audit and feedback has not consistently been found to be effective.³ For every success story there is a project that has run into the ground without showing any substantial contribution to quality of services. The most frequently cited barrier to successful clinical audit is the failure of organisations to provide sufficient protected time for healthcare teams to participate.⁴

The most recent systematic review concluded that providing healthcare professionals with data about their performance in the form of audit and feedback may help improve their practice.⁵ However, in the trials included in the review, the effects varied widely, from an apparent negative to a very large positive effect. When effective, the effects were mainly small to moderate. The review concluded that the relative effectiveness of audit and feedback was likely to be greater when baseline adherence to recommended practice was low and when feedback was delivered more intensively. However, the evidence presented in this review did not support mandatory use of audit and feedback as an intervention to change practice.⁵

It seems ironic that the tool used to measure whether we are doing the right thing in the right way—where audit criteria should be derived from the best evidence—is not itself supported by particularly strong evidence. So why do we persist in audit?

As audit is part of clinical governance, to assess whether patients are receiving the best quality of care it is essential that we measure practice to know when we need to change it. Audit provides us with the best available tool to achieve this objective. An example of a success story is the national audit of stroke in the United Kingdom (excluding Scotland), which has improved quality of care for stroke patients across all three countries.⁶

Audit is included in the foundation programme to allow you, as junior doctors, to gain an understanding of how to obtain, maintain, and improve the services you deliver now and in the future. Performing an audit may also help you in your own learning and understanding of the healthcare process in a particular field. It may also allow you to contribute to constructing or refining a clinical protocol.

Barriers to successful audit may have been a reason for the lack of effect of audit found in some of the trials included in the review mentioned above. So we all need to try to overcome these barriers when we perform an audit.

How can you conduct an audit so that it is a success rather than a failure?

If you are aware of the most frequent impediments to a successful audit, you can try to avoid them and make your audit a success rather than a failure.

Good quality evidence on how to conduct an audit is lacking. However, the two elements that have been shown to be most effective in conducting an audit, based on evidence from a review⁷ and an opinion piece,⁸ are: an environment where audit is made a priority by the trust board, so that it is encouraged and supported; and the existence of a structured programme for audit, where a trust has a central clinical audit office that coordinates audit activity and brings together the results of audit for the trust as a whole. If these elements are not present, the audit is less likely to be a success.

Most healthcare trusts should have a clinical audit lead. This person is responsible for creating and implementing a strategy for clinical audit, setting priorities for audit, and implementing an audit programme.

Data for an audit are generally collected retrospectively. However, prospective data collection can give a team immediate feedback on its current performance and act as positive reinforcement to improve or maintain practice. Prospective audit usually requires good information technology resources.^{4 9 10}

The audit cycle and spiral

Clinical audit can be described as a cyclical or spiral systematic process (figure), with the ultimate aim of improving care. The spiral suggests that as the process continues, each cycle aspires to a higher level of quality.

The audit cycle

Stage 1: Preparing for the audit

Identify problem and local resources for audit

Selecting a topic for audit depends on the objectives of the audit and is likely to involve measuring adherence to healthcare processes that have been shown to produce best outcomes for patients. Consider also incorporating the views of all healthcare professionals involved in patient care, as in the national stroke audit.^{4 5}

The clinical team has an important role in prioritising clinical topics. The following questions may help you select a topic:

• Is the topic a priority for the organisation? For example, have problems been encountered in any of the following areas?

  • -High volume—such as requests for chest x rays in the accident and emergency department

  • -High risk to staff—such as needle stick injuries in an HIV unit

  • -High risk to patients—such as certain postoperative complications

  • -High cost—such as trastuzumab for breast cancer.

• Have patients recommended topics? Patients’ priorities can differ markedly from those of clinicians. Practical approaches have been developed for involving patients in all stages of audit (including design), data collection, and implementing change.^{11 12}

• Is good evidence available to inform standards—for example, systematic reviews or national clinical guidelines?

Locate relevant information

• Where can you find clinical guidelines? From the National Institute for Health and Clinical Excellence (www.nice.org.uk); National Library for Health (www.library.nhs.uk); and Scottish Intercollegiate Guidelines Network (www.sign.ac.uk).

• Where can you find criteria for clinical audit? From clinical guidelines or local hospital guidelines.

• Where can you find information on service standards?

  • -From national service frameworks, which are long term strategies determined by various stakeholders—such as health professionals, service users, and managers—to improve specific areas of care by setting measurable goals within set time frames (www.dh.gov.uk/en/Sitemap/DH_A-Z_AZSI). For example, the goal that no patient should wait longer than one month from an urgent referral by their GP with suspected cancer, to the start of treatment.¹³

  • -From the National Centre for Health Outcomes Development (http://nchod.uhce.ox.ac.uk)

  • -From the Health Commission Wales (Specialist Services) (http://new.wales.gov.uk/topics/health/hcw/?lang=en).

• Which organisations have information about clinical audit? The royal colleges and other professional bodies. The Clinical Effectiveness and Evaluation Unit of the Royal College of Physicians (www.rcplondon.ac.uk/college/ceeu/index.htm) works to promote clinical standards through audits, guidelines, and related activities to improve health care.

Stage 2: Selecting audit review criteria

You can use recommendations from clinical practice guidelines to develop criteria and standards. This could save you time and additional work.

Determine what you are trying to measure

Audit criteria are explicit statements defining an outcome to be measured. They should relate to important aspects of care and be derived from the best available evidence. Having explicit selection criteria will ensure that the data you collect are precise and that you collect only essential information.

Define ideal standards

For the criterion to be useful, you need to define the standard (the level of care to be achieved for any particular criterion, which is usually expressed as a percentage).¹⁴ Ensure that the standard you choose is realistic for your given environment (see the box with a worked example).

Anderson, in his ABC of Audit, writes: “A minimum standard describes the lowest acceptable standard of performance. Minimum standards are often used to distinguish between acceptable and unacceptable practice. An ideal standard describes the care it should be possible to give under ideal conditions, with no constraints. Such a standard by definition cannot usually be attained. An optimum standard lies between the minimum and the ideal. Setting an optimum standard requires judgment, discussion and consensus with other members of the team. Optimum standards represent the standard of care most likely to be achieved under normal conditions of practice.”¹⁵

Two examples of audit criteria (with a different standard defined for each criterion) are:

• To measure what percentage of patients with septic shock were given anti-infective treatment. As septic shock is a condition with a high mortality rate, it would be appropriate to aim for a standard of 100%.

• To measure what percentage of patients in the rheumatology outpatient clinic were seen within one hour. Here it may be acceptable for the standard for the first audit to be 80%.

Stage 3: Measuring levels of performance

Collect data

Some hospitals have audit teams that may help with data collection. An audit proforma is useful for collecting data and can be readily derived from established guidelines and protocols. You need to define the patients to be included and excluded in the audit, the audit review criteria, and the time period over which the criteria apply.

The data may be available in a computerised information system, but it may also be appropriate to collect data manually depending on the outcome being measured. In either case, you will need to consider what data you need to collect, where you will find the data, and who will collect the data.

Although clinical records are frequently used as the source of data, they are often incomplete. Collecting data from several sources—such as clinical records, blood results from patient administration systems, and imaging from picture archiving and communications systems—can help to overcome this problem.

Electronic information systems are useful not only for collecting data but also for improving access to research evidence, prompting change through record templates, and introducing revised systems of care.

Compare performance with criteria

This is the analysis stage.

• Compare the data collected with criteria and standards

• Conclude how well the standards were met

• If they were not met, identify reasons for this.

In theory, if the standard was not met in 100% of the standard that was set, there is potential for improving care. Remember, the standard set may have been 90%. In practice, if the results are close to 100% of the standard, you may decide that any further improvement will be difficult to achieve and that other standards, with results further away from 100%, are the priority targets for action. However, this decision also depends on the topic—in some life threatening situations, it will be important to achieve 100% of the standard.

Stage 4: Making improvements

Implement change

Data collection has no chance of making any impact unless you follow it up with the more difficult process of implementing changes. If you have completed stages 1 to 3 early in your four month attachment, you will be able to present your results within this time period.

Once you have presented and discussed the audit results with the rest of the team and whoever else in the hospital the audit is relevant to, you must agree on recommendations for change. Use an action plan to record these recommendations, also indicating who has agreed to do what and by when.

Disseminating educational materials, such as guidelines, has little effect unless accompanied by selected implementation methods, such as tutorials, reviews, or reminders.⁴

Stage 5: Sustaining improvements

This stage is critical to the successful outcome of an audit: it verifies whether the changes implemented have had an effect and determines whether further improvements are needed to achieve the standards identified in stage 2.

Repeat the audit

Although as a foundation doctor you will not be able to complete the full audit cycle within your four month attachment, you will be able to collect and compare the data within this time period. However, to complete the cycle, after an agreed period, the audit needs to be repeated.

If you wish to complete the audit cycle yourself, you could come back and complete the cycle. If not, you or your consultant must ensure that the cycle is completed by someone else otherwise your time spent on stages 1 to 3 of the audit cycle will have been wasted.

You or whoever is going to complete the audit cycle (possibly the next doctor on the rotation) should use the same strategies for doing the audit to ensure the original audit is comparable. The repeat audit will hopefully show that changes have been implemented and improvements made.

Develop tools to sustain improvements

If these improvements are sustained, some form of monitoring should replace a full audit. The team should develop structures and systems that integrate, monitor, and sustain the improvements implemented as part of clinical audit. But if performance deteriorates, the full audit should be reactivated.

What are the challenges?

The main challenge is how to make your audit a success. Firstly, you need support. Hopefully your trust will have a clinical audit lead; if not, ensure that your consultant is on your side and enthusiastic, otherwise achieving success may be difficult. Tell your consultant about a local audit presentation you attended where a repeat audit showed that patient care really had improved after the audit or cite an audit success story, such as the national stroke audit.

Secondly, give yourself enough time—you should probably start thinking about choosing an audit topic one month into your attachment. With access to a computer and case notes, in a retrospective audit you could collect the data within hours.

Thirdly, be realistic when setting the standards for your audit. Don’t choose a standard of 100% if you know that your trust cannot possibly meet this ideal standard. Aim for an optimum rather than an ideal standard—discuss with the team what the optimum standard should be.

Finally, remember that audits are more likely to be effective when baseline adherence to recommended practice is low and feedback is delivered more intensively.⁵

Worked example: how to perform the perfect audit

Stage 1: Preparing for audit

Identify the problem and the local resources available for audit

• Dr Black is a foundation year 2 doctor on a respiratory firm. One month into his attachment, he has noticed that patients with chronic obstructive airways disease who met the criteria for starting non-invasive ventilation had not been given this treatment. He discusses this with his consultant, who encourages him to do an audit.

• Dr Black discovers that his hospital has a local audit lead and a clinical audit office. He contacts them and is told that they will be able to help him with the audit.

• In agreement with his local audit lead, he chooses the British Thoracic Society guidelines* on non-invasive ventilation in acute respiratory failure and finds these on the society’s website.

Stage 2: Selecting criteria

Determine what you are trying to measure and define gold standards

• Dr Black writes the following criterion statement, derived from the thoracic society’s guidelines: To measure the percentage of patients with an acute exacerbation of chronic obstructive pulmonary disease with a respiratory acidosis (pH <7.35), despite maximum medical treatment on controlled oxygen therapy, who received treatment with non-invasive ventilation.

• After discussion with the respiratory team, he chooses a standard of 90% as he thinks that in the given environment, as an optimum standard, this percentage is more realistic than an ideal standard of 100%.

Stage 3: Measuring level of performance

Collect data

• He decides to collect data from the medical notes of the last 50 patients admitted with an acute exacerbation of chronic obstructive pulmonary disease. He asks the audit office to obtain the notes for him. He creates the following audit proforma (then it takes him two afternoons to collect the data):

  • 1. Does the patient have a respiratory acidosis (pH <7.35) despite maximum medical treatment on controlled oxygen therapy? Yes/No

  • 2. Were they treated with non-invasive ventilation? Yes/No

  • 3. If they were not treated with non-invasive ventilation, was a reason given? Yes/No

Compare performance with criteria

• Dr Black analyses the data against the criterion statement. Forty of the 50 patients had a respiratory acidosis, despite maximum medical treatment on controlled oxygen therapy; 31 of these 40 patients (77.5%) were treated with non-invasive ventilation.

• This percentage did not meet the standard of 90%, so he looked at the reasons why non-invasive ventilation had not been started in nine of these patients. In four patients the respiratory acidosis was noted but it was documented to continue medical treatment. In two patients the respiratory acidosis was thought to be a metabolic acidosis. In two patients no non-invasive ventilation machine was available, and both patients required intubation. In one patient, by the time the decision to start non-invasive ventilation had been made by the medical registrar, the patient had required intubation.

Stage 4: Making improvements

• Three months into his attachment, Dr Black presents the audit to the medical and emergency departments. The consensus is that the two main reasons that non-invasive ventilation was not used were lack of enough machines and lack of knowledge of medical staff on when to use it.

• The clinical directors of the medical and emergency departments agree on the following action plan to improve the use of non-invasive ventilation in these patients:

  • 1. Educate all medical staff on the interpretation of arterial blood gases and use of non-invasive ventilation. This education was to be delivered by tutorials on blood gas interpretation for all junior doctors twice yearly and training in non-invasive ventilation. The respiratory consultants and specialist registrars were to give these in the first week for all new starters.

  • 2. Display posters on when to use non-invasive ventilation in patients with chronic obstructive pulmonary disease in the emergency departments and medical wards. The posters were to be designed by the clinical director of the emergency department and produced by the medical illustration department. The plan was to display them within the next three months.

  • 3. Develop a business plan to increase the number of non-invasive ventilation machines in the hospital. This was to be drafted by the clinical directors of the medical and emergency departments and presented to the chief executive within the next three months.

Stage 5: Sustaining improvements

Repeat audit

• After a year, although Dr Black is now in gastroenterology, he repeats the audit. The first two changes on the action plan have been implemented successfully, and the hospital now has three more non-invasive ventilation machines. He finds that 90% of patients are meeting the criteria.

• Owing to the improvements, Dr Black and the respiratory consultant devise a monitoring tool to ensure that the improvements are sustained. The tool consists of a short checklist on an A4 sheet of paper. This checklist is attached to the notes of all patients with chronic obstructive airways disease. It provides a simple way to ensure that when such patients meet the criteria for treatment with non-invasive ventilation, this treatment is used appropriately. The tool can be used for retrospective or prospective data collection.

• At the repeat audit, it may be appropriate to set the standard higher than 90%.

* The society plans to update this guideline, with work to update it starting in 2008

Summary of elements of effective clinical audit*

• Clinical audit should assess structure, process, or outcomes of care

• The audit should be part of a structured programme and should have a local lead

• Audit should ideally be multidisciplinary

• Patients should ideally be part of the audit

• Choose audit topics based on high risk, high volume, or high cost problems or on national clinical audits, national service frameworks, or NICE guidelines

• Derive standards from good quality guidelines

• Use action plans to overcome the local barriers to change, and identify those responsible for service improvement

• Repeat the audit to find out whether improvements in care have been implemented as a result of clinical audit

• Develop specific mechanisms and systems to monitor and sustain service improvements once the audit cycle has been completed

*Based on A Practical Handbook for Clinical Audit¹⁶

Tips for busy clinicians

• Before you start an audit, find out whether your hospital has a local audit lead. If it does, contact him or her for advice and information

• Choose a topic that reflects national clinical audits, national service frameworks, or NICE guidelines

• Use an audit proforma. This will make collecting the necessary data easier