Using data for improvement

What you need to know.

• Both qualitative and quantitative data are critical for evaluating and guiding improvement

• A family of measures, incorporating outcome, process, and balancing measures, should be used to track improvement work

• Time series analysis, using small amounts of data collected and displayed frequently, is the gold standard for using data for improvement

We all need a way to understand the quality of care we are providing, or receiving, and how our service is performing. We use a range of data in order to fulfil this need, both quantitative and qualitative. Data are defined as “information, especially facts and numbers, collected to be examined and considered and used to help decision-making.”¹ Data are used to make judgements, to answer questions, and to monitor and support improvement in healthcare (box 1). The same data can be used in different ways, depending on what we want to know or learn.

Box 1. Defining quality improvement² .

Quality improvement aims to make a difference to patients by improving safety, effectiveness, and experience of care by:

1. Using understanding of our complex healthcare environment

2. Applying a systematic approach

3. Designing, testing, and implementing changes using real-time measurement for improvement

Within healthcare, we use a range of data at different levels of the system:

• Patient level—such as blood sugar, temperature, blood test results, or expressed wishes for care)

• Service level—such as waiting times, outcomes, complaint themes, or collated feedback of patient experience

• Organisation level—such as staff experience or financial performance

• Population level—such as mortality, quality of life, employment, and air quality.

This article outlines the data we need to understand the quality of care we are providing, what we need to capture to see if care is improving, how to interpret the data, and some tips for doing this more effectively.

Sources and selection criteria.

This article is based on my experience of using data for improvement at East London NHS Foundation Trust, which is seen as one of the world leaders in healthcare quality improvement. Our use of data, from trust board to clinical team, has transformed over the past six years in line with the learning shared in this article. This article is also based on my experience of teaching with the Institute for Healthcare Improvement, which guides and supports quality improvement efforts across the globe.

What data do we need?

Healthcare is a complex system, with multiple interdependencies and an array of factors influencing outcomes. Complex systems are open, unpredictable, and continually adapting to their environment.³ No single source of data can help us understand how a complex system behaves, so we need several data sources to see how a complex system in healthcare is performing.

Avedis Donabedian, a doctor born in Lebanon in 1919, studied quality in healthcare and contributed to our understanding of using outcomes.⁴ He described the importance of focusing on structures and processes in order to improve outcomes.⁵ When trying to understand quality within a complex system, we need to look at a mix of outcomes (what matters to patients), processes (the way we do our work), and structures (resources, equipment, governance, etc).

Therefore, when we are trying to improve something, we need a small number of measures (ideally 5-8) to help us monitor whether we are moving towards our goal. Any improvement effort should include one or two outcome measures linked explicitly to the aim of the work, a small number of process measures that show how we are doing with the things we are actually working on to help us achieve our aim, and one or two balancing measures (box 2). Balancing measures help us spot unintended consequences of the changes we are making. As complex systems are unpredictable, our new changes may result in an unexpected adverse effect. Balancing measures help us stay alert to these, and ought to be things that are already collected, so that we do not waste extra resource on collecting these.

Box 2. Different types of measures of quality of care.

Outcome measures (linked explicitly to the aim of the project)

• Aim—To reduce waiting times from referral to appointment in a clinic

• Outcome measure—Length of time from referral being made to being seen in clinic

  • Data collection—Date when each referral was made, and date when each referral was seen in clinic, in order to calculate the time in days from referral to being seen

Process measures (linked to the things you are going to work on to achieve the aim)

• Change idea—Use of a new referral form (to reduce numbers of inappropriate referrals and re-work in obtaining necessary information)

• Process measure—Percentage of referrals received that are inappropriate or require further information

  • Data collection—Number of referrals received that are inappropriate or require further information each week divided by total number of referrals received each week

• Change idea—Text messaging patients two days before the appointment (to reduce non-attendance and wasted appointment slots)

• Process measure—Percentage of patients receiving a text message two days before appointment

  • Data collection—Number of patients each week receiving a text message two days before their appointment divided by the total number of patients seen each week

• Process measure—Percentage of patients attending their appointment

  • Data collection—Number of patients attending their appointment each week divided by the total number of patients booked in each week

Balancing measures (to spot unintended consequences)

• Measure—Percentage of referrers who are satisfied or very satisfied with the referral process (to spot whether all these changes are having a detrimental effect on the experience of those referring to us)

  • Data collection—A monthly survey to referrers to assess their satisfaction with the referral process

• Measure—Percentage of staff who are satisfied or very satisfied at work (to spot whether the changes are increasing burden on staff and reducing their satisfaction at work)

  • Data collection—A monthly survey for staff to assess their satisfaction at work

How should we look at the data?

This depends on the question we are trying to answer. If we ask whether an intervention was efficacious, as we might in a research study, we would need to be able to compare data before and after the intervention and remove all potential confounders and bias. For example, to understand whether a new treatment is better than the status quo, we might design a research study to compare the effect of the two interventions and ensure that all other characteristics are kept constant across both groups. This study might take several months, or possibly years, to complete, and would compare the average of both groups to identify whether there is a statistically significant difference.

This approach is unlikely to be possible in most contexts where we are trying to improve quality. Most of the time when we are improving a service, we are making multiple changes and assessing impact in real-time, without being able to remove all confounding factors and potential bias. When we ask whether an outcome has improved, as we do when trying to improve something, we need to be able to look at data over time to see how the system changes as we intervene, with multiple tests of change over a period. For example, if we were trying to improve the time from a patient presenting in the emergency department to being admitted to a ward, we would likely be testing several different changes at different places in the pathway. We would want to be able to look at the outcome measure of total time from presentation to admission on the ward, over time, on a daily basis, to be able to see whether the changes made lead to a reduction in the overall outcome. So, when looking at a quality issue from an improvement perspective, we view smaller amounts of data but more frequently to see if we are improving over time.²

What is best practice in using data to support improvement?

Best practice would be for each team to have a small number of measures that are collectively agreed with patients and service users as being the most important ways of understanding the quality of the service being provided. These measures would be displayed transparently so that all staff, service users, and patients and families or carers can access them and understand how the service is performing. The data would be shown as time series analysis, to provide a visual display of whether the service is improving over time. The data should be available as close to real-time as possible, ideally on a daily or weekly basis. The data should prompt discussion and action, with the team reviewing the data regularly, identifying any signals that suggest something unusual in the data, and taking action as necessary.

The main tools used for this purpose are the run chart and the Shewhart (or control) chart. The run chart (fig 1) is a graphical display of data in time order, with a median value, and uses probability-based rules to help identify whether the variation seen is random or non-random.² The Shewhart (control) chart (fig 2) also displays data in time order, but with a mean as the centre line instead of a median, and upper and lower control limits (UCL and LCL) defining the boundaries within which you would predict the data to be.⁶ Shewhart charts use the terms “common cause variation” and “special cause variation,” with a different set of rules to identify special causes.

Fig 1.

A typical run chart

Fig 2.

A typical Shewhart (or control) chart

Is it just about numbers?

We need to incorporate both qualitative and quantitative data to help us learn about how the system is performing and to see if we improve over time. Quantitative data express quantity, amount, or range and can be measured numerically—such as waiting times, mortality, haemoglobin level, cash flow. Quantitative data are often visualised over time as time series analyses (run charts or control charts) to see whether we are improving.

However, we should also be capturing, analysing, and learning from qualitative data throughout our improvement work. Qualitative data are virtually any type of information that can be observed and recorded that is not numerical in nature. Qualitative data are particularly useful in helping us to gain deeper insight into an issue, and to understand meaning, opinion, and feelings. This is vital in supporting us to develop theories about what to focus on and what might make a difference.⁷ Examples of qualitative data include waiting room observation, feedback about experience of care, free-text responses to a survey.

Using qualitative data for improvement

One key point in an improvement journey when qualitative data are critical is at the start, when trying to identify “What matters most?” and what the team’s biggest opportunity for improvement is. The other key time to use qualitative data is during “Plan, Do, Study, Act” (PDSA) cycles. Most PDSA cycles, when done well, rely on qualitative data as well as quantitative data to help learn about how the test fared compared with our original theory and prediction.

Table 1 shows four different ways to collect qualitative data, with advantages and disadvantages of each, and how we might use them within our improvement work.

Table 1.

Different ways to collect qualitative data for improvement

Data collection method	Advantages	Disadvantages	Using the data
Free-text question in a survey	Quick and easy to create, on paper or electronic	Questions are pre-determined so cannot adapt based on answers Beware of survey fatigue	At the start of a project to capture opinions, ideas, and feedback from service users and staff
Interviews	Can be individual or group Can be structured, semi-structured, or unstructured Can explore deeper meaning	Time intensive Need to facilitate the interview and take notes or record the discussion Analysing large amounts of narrative requires skill	To help us understand the issue we want to work on in more detail with multiple perspectives To help us appreciate a deeper meaning behind people’s views and theories
Observations	Able to see behaviour and impact of human factors in real-world setting Can be useful in understanding robustness of implementation	Time intensive Obtrusive, so risk of Hawthorne (observer) effect—knowing you are being observed affects how you behave	Useful to understand the system from another perspective Can be particularly helpful in monitoring whether implementation has been successful
Review of documents	Large amounts of documentation are usually available, and may yield useful information (such as complaints, incident forms, clinical documentation)	Can be time intensive May need a defined search and sampling strategy—you could ask your informatics or business intelligence team for help	At start of project to identify opportunities for improvement through analysing service user feedback, incidents. or complaints

Tips to overcome common challenges in using data for improvement?

One of the key challenges faced by healthcare teams across the globe is being able to access data that is routinely collected, in order to use it for improvement. Large volumes of data are collected in healthcare, but often little is available to staff or service users in a timescale or in a form that allows it to be useful for improvement. One way to work around this is to have a simple form of measurement on the unit, clinic, or ward that the team own and update. This could be in the form of a safety cross⁸ or tally chart. A safety cross (fig 3) is a simple visual monthly calendar on the wall which allows teams to identify when a safety event (such as a fall) occurred on the ward. The team simply colours in each day green when no fall occurred, or colours in red the days when a fall occurred. It allows the team to own the data related to a safety event that they care about and easily see how many events are occurring over a month. Being able to see such data transparently on a ward allows teams to update data in real time and be able to respond to it effectively.

Fig 3.

Example of a safety cross in use

A common challenge in using qualitative data is being able to analyse large quantities of written word. There are formal approaches to qualitative data analyses, but most healthcare staff are not trained in these methods. Key tips in avoiding this difficulty are (a) to be intentional with your search and sampling strategy so that you collect only the minimum amount of data that is likely to be useful for learning and (b) to use simple ways to read and theme the data in order to extract useful information to guide your improvement work.⁹ If you want to try this, see if you can find someone in your organisation with qualitative data analysis skills, such as clinical psychologists or the patient experience or informatics teams.

Education into practice.

• What are the key measures for the service that you work in?

• Are these measures available, transparently displayed, and viewed over time?

• What qualitative data do you use in helping guide your improvement efforts?

How patients were involved in the creation of this article.

Service users are deeply involved in all quality improvement work at East London NHS Foundation Trust, including within the training programmes we deliver. Shared learning over many years has contributed to our understanding of how best to use all types of data to support improvement. No patients have had input specifically into this article.