Methods, applications, interpretations and challenges of interrupted time series (ITS) data: protocol for a scoping review

Abstract

Objectives

Interrupted time series (ITS) design involves collecting data across multiple time points before and after the implementation of an intervention to assess the effect of the intervention on an outcome. ITS designs have become increasingly common in recent times with frequent use in assessing impact of evidence implementation interventions. Several statistical methods are currently available for analysing data from ITS designs; however, there is a lack of guidance on which methods are optimal for different data types and on their implications in interpreting results. Our objective is to conduct a scoping review of existing methods for analysing ITS data, to summarise their characteristics and properties, as well as to examine how the results are reported. We also aim to identify gaps and methodological deficiencies.

Methods and analysis

We will search electronic databases from inception until August 2016 (eg, MEDLINE and JSTOR). Two reviewers will independently screen titles, abstracts and full-text articles and complete the data abstraction. The anticipated outcome will be a summarised description of all the methods that have been used in analysing ITS data in health research, how those methods were applied, their strengths and limitations and the transparency of interpretation/reporting of the results. We will provide summary tables of the characteristics of the included studies. We will also describe the similarities and differences of the various methods.

Ethics and dissemination

Ethical approval is not required for this study since we are just considering the methods used in the analysis and there will not be identifiable patient data. Results will be disseminated through open access peer-reviewed publications.

Strengths and limitations of this study.

• Interrupted time series (ITS) is the strongest and most commonly utilised quasi-experimental design that is used to assess the effect of an intervention when randomisation is not feasible.

• This review will be the first of its kind to provide a comprehensive overview of the methods available in the analysis of data obtained from ITS designs as well as provide characteristics of these current methods as applied in health research.

• The review is limited to articles published in English language.

• This review study will not assess the quality of the papers or the methods themselves.

Introduction

Quasi-experimental designs are alternative approaches to assess the effect of interventions in healthcare settings when randomisation is impossible, unethical or not feasible due to scarce resources.¹ Quasi-experimental designs are divided into four study design groups namely: (a) quasi-experimental designs without control group; (b) quasi-experimental design with control groups but no pretest; (c) quasi-experimental design with control groups and pretest; and (d) interrupted time series (ITS) designs.²

ITS is a robust quasi-experimental design and is most commonly used in implementation science to evaluate the effect of interventions such as quality improvement programmes or policy changes.^2–4 In ITS designs, data are collected before and after the implementation or introduction of an intervention to examine whether the intervention influenced the outcome of interest relative to the underlying secular trend⁵ as well as competing non-contemporaneous interventions.

In recent times, ITS design has been increasingly common with several applications in clinical and health services research. For instance, the ITS method has been used in assessing the effect of a variety of evidence implementation strategies in different settings, which include assessing the effect of prevention programmes, policy changes and quality improvement initiatives in hospitals.^{4 6–9} ITS designs are included in Cochrane reviews done by the Effective Practice and Organisation of Care (EPOC) Review group.¹⁰

There are several models for analysing ITS data such as segmented linear regression analysis and AutoRegressive Integrated Moving Average (ARIMA). Segmented regression or change-point model is likely the most common statistical method for analysing data from ITS design.¹¹ In segmented regression analysis, a piecewise regression is fitted to the data allowing each segment of the time series to exhibit different levels and trends. The effect of the intervention is then examined by comparing the slopes and intercepts for the pre and postintervention phases.¹² A change in the intercept or level constitutes an immediate intervention effect while a change in slope or trend implies an effect that was experienced over time. A change in trend also allows us to measure the sustainability of the impact of the intervention. Furthermore, segmented regression analysis enables analysts to control for other variables, other than the intervention, that can cause a change in level or trend of the outcome of interest.¹²

Despite the various statistical methods that are available for analysing ITS data, it is not clear which methods are optimal for the different data types and how using different analyses may affect results. As such, researchers and particularly, novice users of the ITS design are unaware of the various methods available. Thus, researchers often use methods such as segmented linear regression to analyse their ITS data, which can lead to spurious results and consequently unreliable conclusions about the effect of interventions. Commonly used methods for analysing ITS data are often inappropriate when the data are aggregated per time point or the data are from a skewed distribution, and are often susceptible to aggregation bias, imprecision and loss of power.^{4 8} It is therefore imperative that the various methods available for analysing ITS data are identified to inform researchers of which methods are available for different data types and to inform future research such that the limitations and gaps of the current methods can be adequately addressed.

Our goal is thus to conduct a scoping review to identify the available methods used in ITS analysis, compare the methods in terms of their strengths and limitations and evaluate the application of the methods as well as identify the methodological gaps with the purpose of addressing the gaps. We aim to:

1. Identify and describe available methods used in ITS data analysis.

2. Describe the similarities and differences between the methods.

3. Describe the strengths and limitations of each method.

4. Provide a review of the application of the methods, that is, how they are used, where they are used and the information they provide.

5. Examine how results are reported and provide a comprehensive description of their interpretation.

6. Identify gaps in methodology and provide direction for future research.

Methods and analysis

We will conduct a scoping review using the methods proposed by Arksey and O’Malley¹³ and the Joanna Briggs Handbook for conducting systematic scoping reviews.¹⁴ We will first perform a systematic literature search for health research studies that employed ITS design and then use the methodological framework of Arksey and O’Malley to facilitate the identification of gaps and methodological deficiencies in the existing literature.

Search strategy

For this study, an experienced information scientist helped to develop the search strategy which will be used to search electronic databases including MEDLINE, JSTOR, PUBMED, EMBASE, CINAHL, Web of Science, PsycINFO and the Cochrane Library from inception until August 2016 for relevant articles. Our electronic search will be supplemented by searching for published conference abstracts from relevant statistical conferences and contacting methodological experts in the field of ITS to identify difficult-to-locate or unpublished material. A draft of the literature search for MEDLINE can be found in online supplementary appendix A. The search strategy will be modified accordingly when necessary for other databases. Additionally, references of the included studies will be scanned for relevant articles.

Eligibility criteria

All studies that report the development, application or comparison of methods employed in the analysis of ITS data will be included. To be included in the analysis, studies must have at least three time points before and after the intervention, have a clearly defined point in time when the intervention was implemented or a definition of the time within which the intervention was rolled out since most interventions are rolled out over a period. Also, the study outcomes must be measured objectively. This criteria are in accordance with the EPOC Cochrane Group definition of an ITS design.¹⁰ There will be no restrictions on the publication status. We will exclude ITS studies that are not related to health research. We define health based on the WHO definition, that is, complete mental, physical and social well-being (http://www.who.int/about/definition/en/print.html). Studies that were written in languages other than English will be excluded due to resource limitations. Review papers will be excluded but their references will be scanned for relevant articles.

Study selection

We will import search results, screen citations and full-text articles into Endnote V.X6 desktop. A calibration exercise will be held prior to the commencement of screening of titles and abstracts to ensure reliability. For these pilot runs, each reviewer will screen a random sample of 50 citations for inclusion. Inter-rater agreement for study inclusion will then be calculated and a percentage agreement value >80% will imply we can commence screening. A value <80% will imply poor agreement and hence the eligibility criteria will be modified and clarified with the reviewers. A second pilot will be run with another random sample of 50 citations. This process will be repeated until we reach an agreement >80% and then the screening will begin. Each title and abstract will be screened independently by two reviewers (CS or EB and JEE) for inclusion using the eligibility criteria (level 1 screening). This will be followed by full-text screening of potentially relevant articles by the reviewers to determine inclusion using the same or modified eligibility criteria (level 2 screening). If the eligibility criteria for level 2 are modified, a training session will be conducted prior to level 2 screening. Conflicts will be resolved by discussion or involvement of a third reviewer (JSH or JB). The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow chart will be used to report study selection. A draft of the eligibility form can be found in online supplementary appendix B.

Data abstraction

General study characteristics such as authorship and year of publication will be abstracted. Other characteristics will be abstracted based on the type of article. For instance, for methodological articles we will obtain the general description of the methods, the type of outcome (eg, continuous, binary) and the similarities and differences of the methods either reported by the authors or based on our own perspective. For articles that report the use of an ITS design/analysis (application articles), data to be abstracted will include the characteristics of the study design (number of intervention phases, eg, pre and postintervention for two phases), the rationale for using an ITS design/analysis, the statistical methods used, assumptions checked and approaches used to summarise the results. Examples used in methodological articles will be treated as separate applications and all relevant data will be extracted where necessary. A draft of the data abstraction form is found in online supplementary appendix C. The data abstraction form will be piloted on a random sample of 50 included studies and modified as required. The abstraction process will be completed by JEE in consultations with JSH.

Risk of bias appraisal

Since this is a scoping review, we will not appraise methodological quality or risk of bias of included studies.¹³

Data summary

For the descriptive analysis, information will be summarised based on the type of articles the data were extracted from. For the methods papers, we will narratively describe each of the methods used for analyses such as segmented linear regression and ARIMA, the assumptions made or tested, outcome data type such as continuous or binary, and applicability of the method to different outcomes of interest based on information provided by the authors, the data summarisation process for appropriate ITS analysis as well as the similarities and differences between different methods.

For the application papers, we will summarise: the frequency of use of the method; the kind of data the method is applied to such as audit, and survey data; the setting in which the ITS design was implemented, such as hospitals or communities; and number of interruption times or phases of intervention.

In addition, we will evaluate results interpretation as well as the transparency of reporting of findings by assessing the information provided by the authors. This will include information on the different ways estimates obtained are reported and interpreted. The characteristics of the included studies will be provided using summary tables, and matrix tables will also be created to compare the various ITS methods and the different ways of reporting and interpretation of findings based on how the data were summarised for analysis. Quantitative data such as the number of time points and number of patients per time point will be summarised by medians and IQRs and categorical data such as outcome data type will be summarised by frequencies and percentages. Finally, we will provide a summary of the methodological gaps that will be identified and highlight some potential areas for future research.

Ethics and dissemination

This is a scoping review of completed studies and hence no ethical approval is required. Quasi-experimental designs, such as ITS, have been identified as the best alternative to investigate the effect of an intervention when randomisation is not feasible and has been used several times in implementation science.¹ There have been several limitations that have been identified with the frequently used methods of analysing data from ITS design making them suboptimal.4 5 8 9 For example, when segmented regression is applied to aggregated data where data at each time point are no longer observed but estimated and hence associated with imprecision. Additionally, although several methods exist, it is not clear which of the methods are optimal for different data types. Finding out the various methods available and when each method will be appropriate is necessary to ensure quality analysis of data since using different analyses may lead to different results.

This study will provide detailed information on the available methods used under different scenarios and thus strengthen the validity of research that uses ITS methods. Subsequently, we will identify methodological gaps for future research into various ways of improving ITS data analysis through both simulation and empirical studies. Thus, this review will serve as a groundwork towards developing a guidance manuscript or book on methods for ITS analysis. This review will be the first of its kind to provide a comprehensive overview of the methods available in the analysis of data obtained from quasi-experimental designs as well as provide characteristics of these current methods as applied in health research. Informed consent and ethical assessment was not required for this study since secondary data with non-identifiable individual patient data will be used.

This scoping review will be relevant to statisticians and methodologists and to other stakeholders such as health professionals and policymakers, and most importantly those interested in implementation science.