Interrupted Time Series Analysis for Quality Improvement Projects

Abstract

Quality improvement (QI) projects often use a simple pre/post analysis. While easy and intuitive, this approach ignores underlying temporal trends, thus potentially leading to inaccurate conclusions. Interrupted times series (ITS) analysis takes trends into account and can yield more robust conclusions. This methodological brief provides an overview of time series data and their unique features, discusses the advantages of interrupted time series modeling over simple two-period pre/post approaches, and illustrates the concepts via an exploration of a published example.

Introduction

Broadly speaking, quality improvement (QI) interventions are changes made to a system to improve its performance, aiming to increase the occurrence of a desirable event or decrease the occurrence of an undesirable one. I will refer to both as the “focal indicator.” Medical examples might include efforts to increase colon cancer screening in an office practice, or to decrease the incidence of central line infections in a hospital.

Pre/post designs are commonly used for analysis of QI interventions. A search on the PRiMER journal website for the text string “pre” co-occurring with “post” currently yields 60 results. Searching PubMed for “pre” and “post” and “quality improvement” currently produces nearly 4,000 citations.

The blue vertical bars in Figure 1 illustrate the general outline of a pre/post design:

Figure 1.

General Outline of a Pre/Post Design

The blue vertical bars represent the averages of the focal indicator measurements in their respective pre and post periods. The difference in those measurements suggests the intervention “worked”. But had the focal indicator been measured repeatedly, in shorter time intervals, as represented by the grey dots, an underlying improving trend unrelated to the intervention would have been detected, suggesting that the intervention had no effect.

1. Measure the focal indicator during some arbitrary period of time. This “pre” period is sometimes referred to as a “historical control.”

2. Launch the intervention.

3. Measure the focal indicator again during some arbitrary period of time. This “post” period is often, but needn’t be, the same length as the “pre.”

4. Compare the pre and post measurements, often with some sort of statistical hypothesis.

The measuring, both pre and post, could be done retrospectively (eg, chart review) or prospectively.

Although conceptually simple, this approach has several drawbacks, chief among them that time passes and things change, unrelated to the intervention. Staff, policies, procedures, epidemics, etc inevitably come and go. Any of these could influence the occurrence of the focal indicator and lead to the conclusion, possibly erroneous, that the QI intervention “worked.” Unlike pre/post designs, a time series of measurements can disclose and control for ongoing trends, such as the grey dots in Figure 1. Brady et al provide graphical examples of underlying trends confounding the conclusions drawn from simple pre/post studies.¹

The Nature of Time Series Data

A time series consists of repeated measurements of the same indicator at fixed intervals. Trends and autocorrelation are key features of time series that must be kept in mind during analysis.

Trends

Two common types of trends are secular and seasonal. Secular trends have also been called “rising tides.”² They represent overall long-term improvements independent of the particular QI initiative being implemented. Seasonal trends, such as the epidemiology of influenza, are familiar to physicians. Seasonal trends are also part of residency practices, where we hope that performance improves from July to June each year.

Autocorrelation

Much of classical statistical inference assumes that observations are independent of one another. However, time periods often are not: what happens this week in a medical practice or hospital may depend, to some degree, on what happened last week. This is known as “temporal autocorrelation.” If you know yesterday’s temperature, you have a fair, albeit imperfect, idea of what today’s temperature might be. Before the current period’s measurement is even obtained, something about it is already known, based on previous measurements. An autocorrelated data set contains less information overall than an uncorrelated sample of equal size, because each observation is conveying partially redundant information.

Interpreting a QI Study

The results of a QI study can be interpreted on a variety of levels:

1. If the goal is simply to see if the focal indicator changed, then a rudimentary pre/post analysis (represented by the vertical blue bars in Figure 1) suffices. It allows assertions like, “The frequency of colon cancer screening among eligible patients was higher after we made the change than it was before.”

2. A more ambitious goal would be to associate the timing of the change with that of the intervention, controlling for trends.³ This is where ITS fits in. ITS analysis allows assertions like, “Central line infections started declining when we made the change.”

3. The boldest claim is that the intervention caused the change in the focal indicator. This generally requires a suitable control setting where the intervention was not implemented—a “usual care” setting. This can present significant operational, administrative, financial, and ethical challenges, but examples can be found.^4–6 A staged launch of the QI intervention can sometimes provide a control group.⁷ Rarely, individual patients can be randomized to the QI intervention or control—illustrating the fuzzy distinction between quality improvement and research.⁸

Understanding ITS Model Output

Like most statistical modeling, ITS analysis produces numerical results that represent real-world phenomena. Figure 2 shows the substantive meaning of typical output from a simple ITS model.

Figure 2.

A Pictorial Representation of an Interrupted Time Series Model

The black lines represent the secular trend. It can be upward, flat, or downward, depending on whether B_1, produced by data and software, is positive, zero, or negative, respectively. It can also differ on either side of the intervention. The magnitude of B₁ determines the steepness of the slope. The blue arrow is a level change. It can point upward, be absent, or point downward, depending on whether B₂, produced by data and software, is positive, zero or negative. The magnitude of B₂ determines the length of the arrow.

Interpreting Interrupted Time Series in Uncontrolled QI Studies

Absent a control setting, Bradford Hill’s criteria for assessing possible causation from observational studies can be applied to an ITS analysis.^9,10 When we have implemented an intervention and observed an improvement in a time series of our focal indicator, we are more confident that our intervention might be the cause when:

• The temporal sequence is correct. Improvements already underway cannot stem from the intervention.

• The improvement is large and unmistakable.

• The result is consistent when the intervention is implemented in other settings.

• There is a plausible mechanism, based on sound biopsychosocial or administrative theory. An improvement in the rate of colon cancer screening is unlikely to have been caused by the new coat of paint in the waiting room, whereas improvement in patient satisfaction scores might well have been.

An Example of Interrupted Time Series Analysis

A study by Borde et al, provides an instructive example of ITS analysis that contains both strengths and weaknesses.¹¹ The investigators implemented a multifaceted intervention in the emergency department of a single academic medical center to increase the use of narrower rather than broader-spectrum antibiotics. The authors helpfully provided the equation representing their ITS model:

$$Y_t=B_0+(B_1\times {time}_t)+(B_2\times {intervention}_t)+u_t$$ (1)

u_t represents the inevitable random variation in the system; we will ignore it for now. Table 1 provides the details.

Table 1.

Data Arrangement and Statistical Model in Borde et al

Pre-intervention			Post-intervention
Month and year	t	Intervention	Month and year	t	Intervention
January 2008	0	0	November 2011		1
February 2008	1	0	December 2011		1
:	:	:	:	:	:
September 2011	44	0	September 2013	68	1
October 2011	45	0	October 2013	69	1
Yt = B0 + B1 × timet + B2 × intervention Yt = B0 + B1 × timet + B2 × 0 Yt = B0 + B1 × timet			Yt = B0 + B1 × timet + B2 × intervention Yt = B0 + B1 × timet + B2 × 1 Yt = B0 + B1 × timet + B2

Time goes by month and is represented by t, starting at zero in January 2008 and ending at 69 in October 2013. Y_t is some measure of the use of an antibiotic class in any given month t. The value of intervention is 0 in every pre-intervention month an is 1 in every post-intervention month. Thus B₀ is antibiotic use in January 2008 when t equals zero. B₁ represents the straight-line trend—the monthly change in antibiotic use, averaged across the entire study period; notice that B₁ appears in the same way in the equations for both the pre- and post-intervention periods. B₂ represents what Borde, et al are testing for: a sudden change (upshift or downshift) in use of a particular antibiotic class upon launching the intervention in November 2011 (t = 46), and lasting throughout the entire post-intervention period (t = 46 through t = 69). Notice that B₂ disappears from the equation in the pre-intervention period, because it is multiplied by zero.

We will consider only their analysis of first- and second-generation cephalosporin use, which they hoped to preferentially increase. They hypothesized an abrupt increase in use upon implementing the intervention—a “level change.” In terms of the model represented by Equation 1, this is B₂, which statistical software will calculate based on sample data. B₂ significantly greater than zero would suggest that the intervention worked—the use of first- and second-generation cephalosporins increased abruptly when it was launched. Conversely, an estimate of B₂ that could plausibly be zero would suggest that it did not.

Borde et al controlled for a secular trend—that is B₁ in the model equation. They also controlled for autocorrelation (details not discussed here).

However, their model includes two consequential assumptions, or constraints, regarding the secular trend: (1) that it is the same before and after the intervention, and (2) that it is a straight line. Neither of these assumptions is required, and in this example the conclusion—that there was an abrupt increase in first- and second-generation cephalosporin use—is likely an artifact of those two assumptions, as shown in Figure 3.

Figure 3.

Example of an Interrupted Time-Series Analysis Comparing Preintervention and Postintervention Data

Black lines are the observed monthly data. The author’s interest was in whether there was an abrupt change in antibiotic class use when they implemented their intervention in November 2011—that is, whether B₂, the length of the vertical blue arrow, was non-zero. Their model imposed assumptions or constraints that the two red lines—the pre and post trends—had to be parallel and had to be straight. This forced the analysis to produce a non-zero B₂. Underlying figure reproduced from: Borde JP, Kern VW, Hug M, et al. Implementation of an intensified antibiotic stewardship programme targeting third-generation cephalosporin and fluoroquinolone use in an emergency medicine department. Emergency Medicine Journal. 2015;32:509–515. Adapted with permission from BMJ Publishing Group Ltd.

Relaxing the first assumption and allowing the slope of the trend line to differ before and after the intervention fits the observed data much better. This model would have led to a different conclusion: that there was no level change, but rather a dramatic reversal of what had been a gradual decrease in the use of the narrower-spectrum agents. Relaxing the second assumption and allowing trends to be curved would again have shown no level change, but rather a reversal in the decreasing use of narrower-spectrum agents beginning a year before the intervention (Figure 4). While perhaps just as clinically gratifying, these are different conclusions from those reached by the authors.

Figure 4.

Relaxing Linearity Contraints Produces a Better Model and Leads to Different Conclusions About the Effect of the Intervention

Black lines are the observed monthly data. The author’s interest was in whether there was an abrupt change in antibiotic class use when they implemented their intervention in November 2011. The red line is the result of changing the ITS model in Figure 3 to allow dierent trend slopes before and after the intervention. B₂, representing a “level shift”, is no longer necessary and disappears (becomes zero). The interpretation would be that there was no sudden level shift, but rather that a trend of decreasing use of narrower-spectrum agents reversed around the time of the intervention, and their use increased dramatically over the ensuing two years. The broken purple curve is the result of changing the ITS model to allow non-linear trends. Again B₂, representing a level shift, is no longer needed so is not apparent. This approach suggests that monthly use of narrower-spectrum agents may have started to accelerate about a year before the intervention. Underlying figure reproduced from: Borde JP, Kern WV, Hug M, et al. Implementation of an intensified antibiotic stewardship programme targeting third-generation cephalosporin and fluoroquinolone use in an emergency medicine department. Emergency Medicine Journal. 2015;32:509–515. Reproduced with permission from BMJ Publishing Group Ltd.

Summary

The passage of time is inevitable and can affect outcomes of interest. Pre/post QI studies do not adequately control for trends arising from the passage of time. Interrupted time series modeling, the QI intervention being the “interruption,” provides a more robust, informative, and flexible approach.

The ITS example explored in detail here serves also as a reminder that all statistical tests and models entail assumptions, some of them chosen by the analysts. To interpret findings, it is important to understand the constraints imposed by the model used. In ITS, it is generally best to allow for the possibility of nonlinear trends that differ pre- and post-intervention.