Single-Case Design, Analysis, and Quality Assessment for Intervention Research

Abstract

Background and Purpose

The purpose of this article is to describe single-case studies, and contrast them with case studies and randomized clinical trials. We will highlight current research designs, analysis techniques, and quality appraisal tools relevant for single-case rehabilitation research.

Summary of Key Points

Single-case studies can provide a viable alternative to large group studies such as randomized clinical trials. Single case studies involve repeated measures, and manipulation of and independent variable. They can be designed to have strong internal validity for assessing causal relationships between interventions and outcomes, and external validity for generalizability of results, particularly when the study designs incorporate replication, randomization, and multiple participants. Single case studies should not be confused with case studies/series (ie, case reports), which are reports of clinical management of one patient or a small series of patients.

Recommendations for Clinical Practice

When rigorously designed, single-case studies can be particularly useful experimental designs in a variety of situations, even when researcher resources are limited, studied conditions have low incidences, or when examining effects of novel or expensive interventions. Readers will be directed to examples from the published literature in which these techniques have been discussed, evaluated for quality, and implemented.

Introduction

The purpose of this article is to present current tools and techniques relevant for single-case rehabilitation research. Single-case (SC) studies have been identified by a variety of names, including “n of 1 studies” and “single-subject” studies. The term “single-case study” is preferred over the previously mentioned terms because previous terms suggest these studies include only one participant. In fact, as will be discussed below, for purposes of replication and improved generalizability, the strongest SC studies commonly include more than one participant.

A SC study should not be confused with a “case study/series “ (also called “case report”. In a typical case study/series, a single patient or small series of patients is involved, but there is not a purposeful manipulation of an independent variable, nor are there necessarily repeated measures. Most case studies/series are reported in a narrative way while results of SC studies are presented numerically or graphically.^1,2 This article defines SC studies, contrasts them with randomized clinical trials, discusses how they can be used to scientifically test hypotheses, and highlights current research designs, analysis techniques, and quality appraisal tools that may be useful for rehabilitation researchers.

In SC studies, measurements of outcome (dependent variables) are recorded repeatedly for individual participants across time and varying levels of an intervention (independent variables).^1–5 These varying levels of intervention are referred to as “phases” with one phase serving as a baseline or comparison, so each participant serves as his/her own control.² In contrast to case studies and case series in which participants are observed across time without experimental manipulation of the independent variable, SC studies employ systematic manipulation of the independent variable to allow for hypothesis testing.^1,6 As a result, SC studies allow for rigorous experimental evaluation of intervention effects and provide a strong basis for establishing causal inferences. Advances in design and analysis techniques for SC studies observed in recent decades have made SC studies increasingly popular in educational and psychological research. Yet, the authors believe SC studies have been undervalued in rehabilitation research, where randomized clinical trials (RCTs) are typically recommended as the optimal research design to answer questions related to interventions.⁷ In reality, there are advantages and disadvantages to both SC studies and RCTs that should be carefully considered in order to select the best design to answer individual research questions. While there are a variety of other research designs that could be utilized in rehabilitation research, only SC studies and RCTs are discussed here because SC studies are the focus of this article and RCTs are the most highly recommended design for intervention studies.⁷

When designed and conducted properly, RCTs offer strong evidence that changes in outcomes may be related to provision of an intervention. However, RCTs require monetary, time, and personnel resources that many researchers, especially those in clinical settings, may not have available.⁸ RCTs also require access to large numbers of consenting participants that meet strict inclusion and exclusion criteria that can limit variability of the sample and generalizability of results.⁹ The requirement for large participant numbers may make RCTs difficult to perform in many settings, such as rural and suburban settings, and for many populations, such as those with diagnoses marked by lower prevalence.⁸ To rely exclusively on RCTs has the potential to result in bodies of research that are skewed to address the needs of some individuals while neglecting the needs of others. RCTs aim to include a large number of participants and to use random group assignment to create study groups that are similar to one another in terms of all potential confounding variables, but it is challenging to identify all confounding variables. Finally, the results of RCTs are typically presented in terms of group means and standard deviations that may not represent true performance of any one participant.¹⁰ This can present as a challenge for clinicians aiming to translate and implement these group findings at the level of the individual.

SC studies can provide a scientifically rigorous alternative to RCTs for experimentally determining the effectiveness of interventions.^1,2 SC studies can assess a variety of research questions, settings, cases, independent variables, and outcomes.¹¹ There are many benefits to SC studies that make them appealing for intervention research. SC studies may require fewer resources than RCTs and can be performed in settings and with populations that do not allow for large numbers of participants.^1,2 In SC studies, each participant serves as his/her own comparison, thus controlling for many confounding variables that can impact outcome in rehabilitation research, such as gender, age, socioeconomic level, cognition, home environment, and concurrent interventions.^2,11 Results can be analyzed and presented to determine whether interventions resulted in changes at the level of the individual, the level at which rehabilitation professionals intervene.^2,12 When properly designed and executed, SC studies can demonstrate strong internal validity to determine the likelihood of a causal relationship between the intervention and outcomes and external validity to generalize the findings to broader settings and populations.^2,12,13

Single Case Research Designs for Intervention Research

There are a variety of SC designs that can be used to study the effectiveness of interventions. Here we discuss: 1) AB designs, 2) reversal designs, 3) multiple baseline designs, and 4) alternating treatment designs, as well as ways replication and randomization techniques can be used to improve internal validity of all of these designs.^{1–3,12–14}

The simplest of these designs is the AB Design¹⁵ (Figure 1). This design involves repeated measurement of outcome variables throughout a baseline control/comparison phase (A ) and then throughout an intervention phase (B). When possible, it is recommended that a stable level and/or rate of change in performance be observed within the baseline phase before transitioning into the intervention phase.² As with all SC designs, it is also recommended that there be a minimum of five data points in each phase.^1,2 There is no randomization or replication of the baseline or intervention phases in the basic AB design.² Therefore, AB designs have problems with internal validity and generalizability of results.¹² They are weak in establishing causality because changes in outcome variables could be related to a variety of other factors, including maturation, experience, learning, and practice effects.^2,12 Sample data from a single case AB study performed to assess the impact of Floor Play intervention on social interaction and communication skills for a child with autism¹⁵ are shown in Figure 1.

Figure 1.

An example of results from a single-case AB study conducted on one participant with autism; two weeks of observation (baseline phase A) were followed by seven weeks of Floor Time Play (intervention phase B). The outcome measure Circles of Communications (reciprocal communication with two participants responding to each other verbally or nonverbally) served as a behavioral indicator of the child’s social interaction and communication skills (higher scores indicating better performance). A statistically significant improvement in Circles of Communication was found during the intervention phase as compared to the baseline. Note that although a stable baseline is recommended for SC studies, it is not always possible to satisfy this requirement, as you will see in Figures 1–4. Data were extracted from Dionne and Martini (2011)¹⁵ utilizing Rohatgi’s WebPlotDigitizer software.⁷⁸

If an intervention does not have carry-over effects, it is recommended to use a Reversal Design.² For example, a reversal A₁BA₂ design¹⁶ (Figure 2) includes alternation of the baseline and intervention phases, whereas a reversal A₁B₁A₂B₂ design¹⁷ (Figure 3) consists of alternation of two baseline (A₁, A₂) and two intervention (B₁, B₂) phases. Incorporating at least four phases in the reversal design (i.e., A₁B₁A₂B₂ or A₁B₁A₂B₂A₃B₃…) allows for a stronger determination of a causal relationship between the intervention and outcome variables, because the relationship can be demonstrated across at least three different points in time – change in outcome from A₁ to B₁, from B₁ to A₂, and from A₂ to B₂.¹⁸ Before using this design, however, researchers must determine that it is safe and ethical to withdraw the intervention, especially in cases where the intervention is effective and necessary.¹²

Figure 2.

An example of results from a single-case A₁BA₂ study conducted on eight participants with stable multiple sclerosis (data on three participants were used for this example). Four weeks of observation (baseline phase A₁) were followed by eight weeks of core stability training (intervention phase B), then another four weeks of observation (baseline phase A₂). Forward functional reach test (the maximal distance the participant can reach forward or lateral beyond arm’s length, maintaining a fixed base of support in the standing position; higher scores indicating better performance) significantly improved during intervention for Participants 1 and 3 without further improvement observed following withdrawal of the intervention (during baseline phase A₂). Data were extracted from Freeman et al. (2010)¹⁶ utilizing Rohatgi’s WebPlotDigitizer software.⁷⁸

Figure 3.

An example of results from a single-case A₁B₁A₂B₂ study conducted on two participants with severe unilateral neglect after a right-hemisphere stroke. Two weeks of conventional treatment (baseline phases A_1, A₂) alternated with two weeks of visuo-spatio-motor cueing (intervention phases B₁, B₂). Performance was assessed in two tests of lateral neglect, the Bells Cancellation Test (Figure A; lower scores indicating better performance) and the Line Bisection Test (Figure B; higher scores indicating better performance). There was a statistically significant intervention-related improvement in participants’ performance on the Line Bisection Test, but not on the Bells Test. Data were extracted from Samuel at al. (2000)¹⁷ utilizing Rohatgi’s WebPlotDigitizer software.⁷⁸

A recent study used an ABA reversal SC study to determine the effectiveness of core stability training in 8 participants with multiple sclerosis.¹⁶ During the first four weekly data collections, the researchers ensured a stable baseline, which was followed by eight weekly intervention data points, and concluded with four weekly withdrawal data points. Intervention significantly improved participants’ walking and reaching performance (Figure 2).¹⁶ This A₁BA₂ design could have been strengthened by the addition of a second intervention phase for replication (A₁B₁A₂B₂). For instance, a single-case A₁B₁A₂B₂ withdrawal design aimed to assess the efficacy of rehabilitation using visuo-spatio-motor cueing for two participants with severe unilateral neglect after a severe right-hemisphere stroke.¹⁷ Each phase included 8 data points. Statistically significant intervention-related improvement was observed, suggesting that visuo-spatio-motor cueing might be promising for treating individuals with very severe neglect (Figure 3).¹⁷

The reversal design can also incorporate a cross over design where each participant experiences more than one type of intervention. For instance, a B₁C₁B₂C₂ design could be used to study the effects of two different interventions (B and C) on outcome measures. Challenges with including more than one intervention involve potential carry-over effects from earlier interventions and order effects that may impact the measured effectiveness of the interventions.^2,12 Including multiple participants and randomizing the order of intervention phase presentations are tools to help control for these types of effects.¹⁹

When an intervention permanently changes an individual’s ability, a return to baseline performance is not feasible and reversal designs are not appropriate. Multiple Baseline Designs (MBDs) are useful in these situations (Figure 4).²⁰ MBDs feature staggered introduction of the intervention across time: each participant is randomly assigned to one of at least 3 experimental conditions characterized by the length of the baseline phase.²¹ These studies involve more than one participant, thus functioning as SC studies with replication across participants. Staggered introduction of the intervention allows for separation of intervention effects from those of maturation, experience, learning, and practice. For example, a multiple baseline SC study was used to investigate the effect of an anti-spasticity baclofen medication on stiffness in five adult males with spinal cord injury.²⁰ The subjects were randomly assigned to receive 5–9 baseline data points with a placebo treatment prior to the initiation of the intervention phase with the medication. Both participants and assessors were blind to the experimental condition. The results suggested that baclofen might not be a universal treatment choice for all individuals with spasticity resulting from a traumatic spinal cord injury (Figure 4).²⁰

Figure 4.

An example of results from a single-case multiple baseline study conducted on five participants with spasticity due to traumatic spinal cord injury. Total duration of data collection was nine weeks. The first participant was switched from placebo treatment (baseline) to baclofen treatment (intervention) after five data collection sessions, whereas each consecutive participant was switched to baclofen intervention at the subsequent sessions through the ninth session. There was no statistically significant effect of baclofen on viscous stiffness at the ankle joint. Data were extracted from Hinderer at al. (1990)²⁰ utilizing Rohatgi’s WebPlotDigitizer software.⁷⁸

The impact of two or more interventions can also be assessed via Alternating Treatment Designs (ATDs). In ATDs, after establishing the baseline, the experimenter exposes subjects to different intervention conditions administered in close proximity for equal intervals (Figure 5).²² ATDs are prone to “carry-over effects” when the effects of one intervention influence the observed outcomes of another intervention.¹ As a result, such designs introduce unique challenges when attempting to determine the effects of any one intervention and have been less commonly utilized in rehabilitation. An ATD was used to monitor disruptive behaviors in the school setting throughout a baseline followed by an alternating treatment phase with randomized presentation of a control condition or an exercise condition.²³ Results showed that 30 minutes of moderate to intense physical activity decreased behavioral disruptions through 90 minutes after the intervention.²³ An ATD was also used to compare the effects of commercially available and custom-made video prompts on the performance of multi-step cooking tasks in four participants with autism.²² Results showed that participants independently performed more steps with the custom-made video prompts (Figure 5).²²

Figure 5.

An example of results from a single case alternating treatment study conducted on four participants with autism (data on two participants were used for this example). After the observation phase (baseline), effects of commercially available and custom-made video prompts on the performance of multi-step cooking tasks were identified (treatment phase), after which only the best treatment was used (best treatment phase). Custom-made video prompts were most effective for improving participants’ performance of multi-step cooking tasks. Data were extracted from Mechling at al. (2013)²² utilizing Rohatgi’s WebPlotDigitizer software.⁷⁸

Regardless of the SC study design, replication and randomization should be incorporated when possible to improve internal and external validity.¹¹ The reversal design is an example of replication across study phases. The minimum number of phase replications needed to meet quality standards is three (A₁B₁A₂B₂), but having four or more replications is highly recommended (A₁B₁A₂B₂A₃…).^11,14 In cases when interventions aim to produce lasting changes in participants’ abilities, replication of findings may be demonstrated by replicating intervention effects across multiple participants (as in multiple-participant AB designs), or across multiple settings, tasks, or service providers. When the results of an intervention are replicated across multiple reversals, participants, and/or contexts, there is an increased likelihood a causal relationship exists between the intervention and the outcome.^2,12

Randomization should be incorporated in SC studies to improve internal validity and the ability to assess for causal relationships among interventions and outcomes.¹¹ In contrast to traditional group designs, SC studies often do not have multiple participants or units that can be randomly assigned to different intervention conditions. Instead, in randomized phase-order designs, the sequence of phases is randomized. Simple or block randomization is possible. For example, with simple randomization for an A₁B₁A₂B₂ design, the A and B conditions are treated as separate units and are randomly assigned to be administered for each of the pre-defined data collection points. As a result, any combination of A-B sequences is possible without restrictions on the number of times each condition is administered or regard for repetitions of conditions (e.g., A₁B₁B₂A₂B₃B₄B₅A₃B₆A₄A₅A₆). With block randomization for an A₁B₁A₂B₂ design, two conditions (e.g., A and B) would be blocked into a single unit (AB or BA), randomization of which to different time periods would ensure that each condition appears in the resulting sequence more than two times (e.g., A₁B₁B₂A₂A₃B₃A₄B₄). Note that AB and reversal designs require that the baseline (A) always precedes the first intervention (B), which should be accounted for in the randomization scheme.^2,11

In randomized phase start-point designs, the lengths of the A and B phases can be randomized.^2,11,24–26 For example, for an AB design, researchers could specify the number of time points at which outcome data will be collected, (e.g., 20), define the minimum number of data points desired in each phase (e.g., 4 for A, 3 for B), and then randomize the initiation of the intervention so that it occurs anywhere between the remaining time points (points 5 and 17 in the current example).^27,28 For multiple-baseline designs, a dual-randomization, or “regulated randomization” procedure has been recommended.²⁹ If multiple-baseline randomization depends solely on chance, it could be the case that all units are assigned to begin intervention at points not really separated in time.³⁰ Such randomly selected initiation of the intervention would result in the drastic reduction of the discriminant and internal validity of the study.²⁹ To eliminate this issue, investigators should first specify appropriate intervals between the start points for different units, then randomly select from those intervals, and finally randomly assign each unit to a start point.²⁹

Single Case Analysis Techniques for Intervention Research

The What Works Clearinghouse (WWC) single-case design technical documentation provides an excellent overview of appropriate SC study analysis techniques to evaluate the effectiveness of intervention effects.^1,18 First, visual analyses are recommended to determine whether there is a functional relation between the intervention and the outcome. Second, if evidence for a functional effect is present, the visual analysis is supplemented with quantitative analysis methods evaluating the magnitude of the intervention effect. Third, effect sizes are combined across cases to estimate overall average intervention effects which contributes to evidence-based practice, theory, and future applications.^2,18

Visual Analysis

Traditionally, SC study data are presented graphically. When more than one participant engages in a study, a spaghetti plot showing all of their data in the same figure can be helpful for visualization. Visual analysis of graphed data has been the traditional method for evaluating treatment effects in SC research.^1,12,31,32 The visual analysis involves evaluating level, trend, and stability of the data within each phase (i.e., within-phase data examination) followed by examination of the immediacy of effect, consistency of data patterns, and overlap of data between baseline and intervention phases (i.e., between-phase comparisons). When the changes (and/or variability) in level are in the desired direction, are immediate, readily discernible, and maintained over time, it is concluded that the changes in behavior across phases result from the implemented treatment and are indicative of improvement.³³ Three demonstrations of an intervention effect are necessary for establishing a functional relation.¹

Within-phase examination

Level, trend, and stability of the data within each phase are evaluated. Mean and/or median can be used to report the level, and trend can be evaluated by determining whether the data points are monotonically increasing or decreasing. Within-phase stability can be evaluated by calculating the percentage of data points within 15% of the phase median (or mean). The stability criterion is satisfied if about 85% (80% – 90%) of the data in a phase fall within a 15% range of the median (or average) of all data points for that phase.³⁴

Between-phase examination

Immediacy of effect, consistency of data patterns, and overlap of data between baseline and intervention phases are evaluated next. For this, several nonoverlap indices have been proposed that all quantify the proportion of measurements in the intervention phase not overlapping with the baseline measurements.³⁵ Nonoverlap statistics are typically scaled as percent from 0 to 100, or as a proportion from 0 to 1. Here, we briefly discuss the Nonoverlap of All Pairs (NAP),³⁶ the Extended Celeration Line (ECL), the Improvement Rate Difference (IRD),³⁷ and the TauU and the TauU-adjusted, TauU_adj,³⁵ as these are the most recent and complete techniques. We also examine the Percentage of Nonoverlapping Data (PND)³⁸ and the Two Standard Deviations Band Method, as these are frequently used techniques. In addition, we include the Percentage of Nonoverlapping Corrected Data (PNCD) – an index applying to the PND after controlling for baseline trend.³⁹

Nonoverlap of all pairs (NAP)

Each baseline observation can be paired with each intervention phase observation to make n pairs (i.e., N = n_A*n_B). Count the number of overlapping pairs, n_o, counting all ties as 0.5. Then define the percent of the pairs that show no overlap. Alternatively, one can count the number of positive (P), negative (N), and tied (T) pairs^2,36:

$$\mathit{NAP}=\frac{n-n_0}{n}\ast 100=\frac{P+0.5T}{P+T+N}\ast 100$$

Extended Celeration Line (ECL)

ECL or split middle line allows control for a positive Phase A trend. Nonoverlap is defined as the proportion of Phase B (n_b) data that are above the median trend plotted from Phase A data (n_{B_{Above Median trend A}}), but then extended into Phase B: $\mathit{ECL}=\frac{n_{B_{\mathit{Above}\mathit{Median}\mathit{trend}A}}}{n_b}\ast 100$

As a consequence, this method depends on a straight line and makes an assumption of linearity in the baseline.^2,12

Improvement rate difference (IRD)

This analysis is conceptualized as the difference in improvement rates (IR) between baseline (IR_B) and intervention phases (IR_T).³⁸ The IR for each phase is defined as the number of “improved data points” divided by the total data points in that phase. IRD, commonly employed in medical group research under the name of “risk reduction” or “risk difference” attempts to provide an intuitive interpretation for nonoverlap and to make use of an established, respected effect size, IR_B - IR_B, or the difference between two proportions.³⁷

TauU and TauU_adj

Each baseline observation can be paired with each intervention phase observation to make n pairs (i.e., n=n_A*n_B). Count the number of positive (P), negative (N), and tied (T) pairs, and use the following formula: $\mathit{TauU}=\frac{P-N}{P+N+\tau }$

The TauU_adj is an adjustment of TauU for monotonic trend in baseline. Each baseline observation can be paired with each intervention phase observation to make n pairs (i.e., n=n_A*n_B). Each baseline observation can be paired with all later baseline observations (n_A*(n_A-1)/2).^2,35 Then the baseline trend can be computed: ${\mathit{TauU}}_{\mathit{adf}}=\frac{P-N-S_{\mathit{trend}}}{P+N+\tau }$; S_trend = P_A – NA

Online calculators might assist researchers in obtaining the TauU and TauU adjusted coefficients (http://www.singlecaseresearch.org/calculators/tau-u).

Percentage of nonoverlapping data (PND)

If anticipating an increase in the outcome, locate the highest data point in the baseline phase and then calculate the percent of the intervention phase data points that exceed it. If anticipating a decrease in the outcome, find the lowest data point in the baseline phase and then calculate the percent of the treatment phase data points that are below it: $\mathit{PND}=\frac{n_{B_{\mathit{Overlap}A}}}{n_b}\ast 100$. A PND < 50 would mark no observed effect, PND = 50–70 signifies a questionable effect, and PND > 70 suggests the intervention was effective.⁴⁰ The percentage of nonoverlapping (PNDC) corrected was proposed in 2009 as an extension of the PND.³⁹ Prior to applying the PND, a data correction procedure is applied eliminating pre-existing baseline trend.³⁸

Two Standard Deviation Band Method

When the stability criterion described above is met within phases, it is possible to apply the two standard deviation band method.^12,41 First, the mean of the data for a specific condition is calculated and represented with a solid line. In the next step, the standard deviation of the same data is computed and two dashed lines are represented: one located two standard deviations above the mean and the other – two standard deviations below. For normally distributed data, few points (less than 5%) are expected to be outside the two standard deviation bands if there is no change in the outcome score due to the intervention. However, this method is not considered a formal statistical procedure, as the data cannot typically be assumed to be normal, continuous, or independent.⁴¹

Statistical Analysis

If the visual analysis indicates a functional relationship (i.e., three demonstrations of the effectiveness of the intervention effect), it is recommended to proceed with the quantitative analyses, reflecting the magnitude of the intervention effect. First, effect sizes are calculated for each participant (individual-level analysis). Moreover, if the research interest lies in the generalizability of the effect size across participants, effect sizes can be combined across cases to achieve an overall average effect size estimate (across-case effect size).

Note that quantitative analysis methods are still being developed in the domain of SC research¹ and statistical challenges of producing an acceptable measure of treatment effect remain.^14,42,43 Therefore, the WWC standards strongly recommend conducting sensitivity analysis and reporting multiple effect size estimators. If consistency across different effect size estimators is identified, there is stronger evidence for the effectiveness of the treatment.^1,18

Individual-level effect size analysis

The most common effect sizes recommended for SC analysis are: 1) standardized mean difference Cohen’s d; 2) standardized mean difference with correction for small sample sizes Hedges’ g; and 3) the regression-based approach which has the most potential and is strongly recommended by the WWC standards.^1,44,45 Cohen’s d can be calculated using following formula: $d=\frac{\overline{X_A}-\overline{X_B}}{s_p}$, with $\overline{X_A}$ being the baseline mean, $\overline{X_B}$ being the treatment mean, and s_p indicating the pooled within-case standard deviation. Hedges’ g is an extension of Cohen’s d, recommended in the context of SC studies as it corrects for small sample sizes. The piecewise regression-based approach does not only reflect the immediate intervention effect, but also the intervention effect across time:

$$Y_i={\beta }_0+{\beta }_1{T}_i+{\beta }_2{D}_i+{\beta }_3{T}_i{D}_i+e_i+\rho e_{i-1}\text{with}{e}_i~N(0,{\sigma }_e^2),$$ (1)

i stands for the measurement occasion (i = 0, 1,…I). The dependent variable is regressed on a time indicator, T, which is centered around the first observation of the intervention phase, D, a dummy variable for the intervention phase, and an interaction term of these variables. The equation shows that the expected score, Ŷ_i, equals β₀ + β₁T_i in the baseline phase, and (β₀ + β₂) + (β₁ + β₃)T_i in the intervention phase. β₀, therefore, indicates the expected baseline level at the start of the intervention phase (when T = 0), whereas β₁ marks the linear time trend in the baseline scores. The coefficient β₂ can then be interpreted as an immediate effect of the intervention on the outcome, whereas β₃ signifies the effect of the intervention across time. The e_i’s are residuals assumed to be normally distributed around a mean of zero with a variance of ${\sigma }_e^2$. The assumption of independence of errors is usually not met in the context of SC studies because repeated measures are obtained within a person. As a consequence, it can be the case that the residuals are autocorrelated, meaning that errors closer in time are more related to each other compared to errors further away in time.^46–48 As a consequence, a lag-1 autocorrelation is appropriate (taking into account the correlation between two consecutive errors: e_i and e_i–1; for more details see Verbeke & Molenberghs, (2000).⁴⁹ In Equation 1, ρ indicates the autocorrelation parameter. If ρ is positive, the errors closer in time are more similar; if ρ is negative, the errors closer in time are more different, and if ρ equals zero, there is no correlation between the errors.

Across-case effect sizes

Two-level modeling to estimate the intervention effects across cases can be used to evaluate across-case effect sizes.^44,45,50 Multilevel modeling is recommended by the WWC standards because it takes the hierarchical nature of SC studies into account: measurements are nested within cases and cases, in turn, are nested within studies. By conducting a multilevel analysis, important research questions can be addressed (which cannot be answered by single-level analysis of SC study data), such as: 1) What is the magnitude of the average treatment effect across cases? 2) What is the magnitude and direction of the case-specific intervention effect? 3) How much does the treatment effect vary within cases and across cases? 4) Does a case and/or study level predictor influence the treatment’s effect? The two-level model has been validated in previous research using extensive simulation studies.^45,46,51 The two-level model appears to have sufficient power (> .80) to detect large treatment effects in at least six participants with six measurements.²¹

Furthermore, to estimate the across-case effect sizes, the HPS (Hedges, Pustejovsky, and Shadish), or single-case educational design (SCEdD)-specific mean difference, index can be calculated.⁵² This is a standardized mean difference index specifically designed for SCEdD data, with the aim of making it comparable to Cohen’s d of group-comparison designs. The standard deviation takes into account both within-participant and between-participant variability, and is typically used to get an across-case estimator for a standardized change in level. The advantage of using the HPS across-case effect size estimator is that it is directly comparable with Cohen’s d for group comparison research, thus enabling the use of Cohen’s (1988) benchmarks.⁵³

Valuable recommendations on SC data analyses have recently been provided.^54,55 They suggest that a specific SC study data analytic technique can be chosen based on: (1) the study aims and the desired quantification (e.g., overall quantification, between-phase quantifications, randomization, etc.), (2) the data characteristics as assessed by visual inspection and the assumptions one is willing to make about the data, and (3) the knowledge and computational resources. ^54,55 Table 1 lists recommended readings and some commonly used resources related to the design and analysis of single-case studies.

Table 1.

Recommend readings and resources related to the design and analysis of single-case studies.

General Readings on Single-Case Research Design and Analysis
Barlow, D.H., Nock, M.K., Hersen, M. Single Case Experimental Designs: Strategies for Studying Behavior Change. 3rd ed. Needham Heights, MA: Allyn & Bacon; 2008.3 Kazdin, A.E. Single-case Research Designs: Methods for Clinical and Applied Settings. New York, NY: Oxford University Press; 2010.4 Kratochwill, T.R., Levin, J.R. Single-case Research Design and Analysis. Hillsdale, NJ: Lawrence Erlbaum Associates; 1992.13 Kratochwill, T.R., Levin, J.R. eds. Single-case Intervention Research: Methodological and Statistical Advances. Washington, D.C.: American Psychological Association; 2014.2 Portney, L. G., Watkins, M. P. Foundations of Clinical Research: Applications to Practice. Philadelphia, PA: F. A. Davis Company; 2015.12
Design
Reversal Design	Feeney, T.J., Ylvisaker, M. Context-sensitive cognitive-behavioral supports for young children with TBI. J Posit Behav Interv. 2008;10(2):115–128.59 Lin, C.Y., Chang, Y.M. Increase in physical activities in kindergarten children with cerebral palsy by employing MaKey-MaKey-based task systems. Res Dev Disabil. 2014;35:1963–1969.60 Samuel, C., Louis-Dreyfus, A., Kaschel, R., et al. Rehabilitation of very severe unilateral neglect by visuo-spatio-motor cueing: Two single case studies. Neuropsychol Rehabil. 2000;10(4):385–399.17
Multiple Baseline Design	Hinderer, S.R., Lehmann, J.F., Price, R., White, O., deLateur, B.J., Deitz, J. Spasticity in spinal cord injured persons: Quantitative effects of baclofen and placebo treatments. Am J Phys Med Rehab. 1990;69(6):311–317.20 Lane-Brown, A., Tate, R. Evaluation of an intervention for apathy after traumatic brain injury: A multiple-baseline, single-case experimental design. J Head Trauma Rehab. 2010;25(6):459–469.61
Alternating Treatment Design	Folino, A., Ducharme, J.M., Greenwald, N. Temporal effects of antecedent exercise on students’ disruptive behaviors: An exploratory study. J School Psychol. 2014;52(5):447–462.23 Mechling, L.C., Atres, K.M., Foster, A.L., Bryant, K.J. Comparing the effects of commercially available and custom-made video prompting for teaching cooking skills to high school students with autism. Rem Spec Ed. 2013; 34(6):371–383.22
Randomization	Kratochwill, T.R., Levin, J.R. Enhancing the scientific credibility of single-case intervention research: Randomization to the rescue. Psychol Methods. 2010;15(2):124–144.11
Analysis
Visual Analysis	Lieberman, L.J., Dunn, J.M., van der Mars, H., McCubbin, J. Peer tutors’ effects on activity levels of deaf students in inclusive elementary physical education. Adapt Phys Act Q. 2000;17(1):20–39.62 Lundblom, E.E.G, Woods, J.J. Working in the classroom: Improving idiom comprehension through classwide peer tutoring. Commun Dis Q. 2012;33(4):202–219.63
Percentage of Nonoverlapping Data (PND)	Banda, D.R., Hart, S.L., Liu-Gitz, L. Impact of training peers and children with autism on social skills during center time activities in inclusive classrooms. Res Autism Spect Dis. 2010;4(4):619–625.64 Oddo, M., Barnett, D.W., Hawkins, R.O., Musti-Roa, S. Reciprocal peer tutoring and repeated reading: Increasing practicality using student groups. Psychol Schools. 2010; 47(8):842–858.65
Nonoverlap of All Pairs (NAP)	Parker, R.I., Vannest, K. An improved effect size for single-case research: Nonoverlap of all pairs. Behav Ther. 2009;40:357–367.36 Peterson-Brown, S., Karich, A.C., Symons, F.J. Examining estimates of effect using non-overlap of all pairs in multiple baseline studies of academic intervention. J Behav Educ. 2012;21(3):203–216.66
Improvement Rate Difference (IRD)	Chen, M., Hyppa-Martin, J.K., Reichle, J.E., Symons, F.J. Comparing single case design overlap-based effect size metrics from studies examining speech generating device interventions. Am J Intell and Dev Disabil. 2016;121(3):169–193. 67 Derakhshandeh, F., Nikmaram, M., Hosseinabad, H.H., et al. Speech characteristics after articulation therapy in children with cleft palate and velopharyngeal dysfunction: A single case experimental design. Am J Ped Otolaryngology. 2016;86:104–113.68
Tau-U/Piecewise Regression	Klingbeil, D., Moeyaert, M., Archerm, C., Chimnoza, T.M., Zwolski, S.A. Examining the efficacy of peer-mediated incremental rehearsal. School Psychol Rev. In press.69 Saddler, K., Saddler, B., Moeyaert, M., Ellis-Robinson, T. Effects of a summarizing strategy on written summaries of children with emotional and behavioral disorders. Rem Spec Educ. 2016 In press. 0.1177/0741932516669051 (2.016).70
HLM	Ingersoll, B., Wainer, A. Initial efficacy of project ImPACT: A parent-mediated social communication intervention for young children with ASD. J Autism Dev Disord. 2013; 43(12): 2943–2952.71 Wade, C.A., Ortiz, C., Gorman, B.S. Two-session group parent training for bedtime noncompliance in head start preschoolers. Child Fam Behav Ther. 2007;29(3):23–55.72

Quality Appraisal Tools for Single-Case Design Research

Quality appraisal tools are important to guide researchers in designing strong experiments and conducting high-quality systematic reviews of the literature. Unfortunately, quality assessment tools for SC studies are relatively novel, ratings across tools demonstrate variability, and there is currently no “gold standard” tool.⁵⁶ Table 2 lists important SC study quality appraisal criteria compiled from the most common scales; when planning studies or reviewing the literature, we recommend readers consider these criteria. Table 3 lists some commonly used SC quality assessment and reporting tools and references to resources where the tools can be located.

Table 2.

Summary of important single-case study quality appraisal criteria.

Criteria	Requirements
1. Design1	The design is appropriate for evaluating the intervention.
2. Method details56	Participants’ characteristics, selection method, and testing setting specifics are adequately detailed to allow future replication.
3. Independent variable1,14,18,56	The independent variable (i.e., the intervention) is thoroughly described to allow replication; fidelity of the intervention is thoroughly documented; the independent variable is systematically manipulated under the control of the experimenter.
4. Dependent variable1,14,18	Each dependent/outcome variable is quantifiable.57 Each outcome variable is measured systematically and repeatedly across time to ensure the acceptable 0.80–0.90 inter-assessor percent agreement (or ≥0.60 Cohen’s kappa) on at least 20% of sessions.73
5. Internal validity1,14,18	The study includes at least three attempts to demonstrate an intervention effect at three different points in time or with three different phase replications.73 Design-specific recommendations: 1) for reversal designs, a study should have ≥4 phases with ≥5 points, 2) for alternating intervention designs, a study should have ≥5 points per condition with ≤2 points per phase, 3) for multiple baseline designs, a study should have ≥6 phases with ≥5 points to meet the WWC standards without reservations18. Assessors are independent and blind to experimental conditions.
6. External Validity14	Experimental effects should be replicated across participants, settings, tasks, and/or service providers.
7. Face Validity1,18,56	The outcome measure should be clearly operationally defined, have a direct unambiguous interpretation, and measure a construct is was designed to measure.
8. Social Validity14,56	Both the outcome variable and the magnitude of change in outcome due to an intervention should be socially important, the intervention should be practical and cost effective.
9. Sample attrition1,18	The sample attrition should be low and unsystematic, since loss of data in SC designs due to overall or differential attrition can produce biased estimates of the intervention’s effectiveness if that loss is systematically related to the experimental conditions.
10. Randomization1,18	If randomization is used, the experimenter should make sure that: 1) equivalence is established at the baseline, and 2) the group membership is determined through a random process.

Table 3.

Quality assessment and reporting tools related to single-case studies.

Quality Assessment & Reporting Tools
What Works Clearinghouse Standards (WWC)	Kratochwill, T.R., Hitchcock, J., Horner, R.H., et al. Institute of Education Sciences: What works clearinghouse: Procedures and standards handbook. http://files.eric.ed.gov/fulltext/ED510743.pdf. Published 2010. Accessed November 20, 2016.1
Quality indicators from Horner et al.	Horner, R.H., Carr, E.G., Halle, J., McGee, G., Odom, S., Wolery, M. The use of single-subject research to identify evidence-based practice in special education. Except Children. 2005;71(2):165–179.14
Evaluative Method	Reichow, B., Volkmar, F., Cicchetti, D. Development of the evaluative method for evaluating and determining evidence-based practices in autism. J Autism Dev Disord. 2008;38(7):1311–1319.74
Certainty Framework	Simeonsson, R., Bailey, D. Evaluating programme impact: Levels of certainty. In: Mitchell, D., Brown, R., eds. Early Intervention Studies for Young Children With Special Needs. London, England: Chapman & Hall; 1991:280–296.75
Evidence in Augmentative and Alternative Communication Scales (EVIDAAC)	Schlosser, R.W., Sigafoos, J., Belfiore, P. EVIDAAC comparative single-subject experimental design scale (CSSEDARS). http://www.evidaac.com/ratings/CSSEDARS.pdf. Published 2009. Accessed November 20, 2016.76
Single-Case Experimental Design (SCED)	Tate, R.L., McDonald, S., Perdices, M., Togher, L., Schulz, R., Savage, S. Rating the methodological quality of single-subject designs and n-of-1 trials: Introducing the Single-Case Experimental Design (SCED) Scale. Neuropsychol Rehabil. 2008;18(4):385–401.6
Logan et al. Scales	Logan, L.R., Hickman, R.R., Harris, S.R., Heriza, C.B. Single-subject research design: Recommendations for levels of evidence and quality rating. Dev Med Child Neurol. 2008;50:99–103.77
Single-Case Reporting Guideline In BEhavioural Interventions (SCRIBE)	Tate, R.L., Perdices, M., Rosenkoetter, U., et al. The Single-Case Reporting guideline In BEhavioural interventions (SCRIBE) 2016 statement. J School Psychol. 2016;56:133–142.57
On-line Resources and Tools
Theory, examples, and tools related to multilevel data analysis	Van den Noortgate, W., Ferron, J., Beretvas, S.N., Moeyaert, M. Multilevel synthesis of single-case experimental data. Katholieke Universiteit Leuven web site. http://ppw.kuleuven.be/home/onderzoek/multilevel-synthesis-of-single-case-experimental-data/.
Tools for computing between-cases standardized mean difference (d-statistic)	Pustejovsky, J.E. scdhlm: A web-based calculator for between-case standardized mean differences (Version 0.2) [Web application]. https://jepusto.shinyapps.io/scdhlm.
Tools for computing NAP, IRD, Tau and other statistics	Vannest, K.J., Parker, R.I., Gonen, O. Single case research: Web based calculators for SCR analysis (Version 1.0) [Web-based application]. College Atation, TX: Texas A&M University. Published 2011. Accessed November 20, 2016. http://www.singlecaseresearch.org/.
Tools for obtaining graphical representations, means, trend lines, PND	Wright, J. Intervention central. Accessed November 20, 2016. http://www.interventioncentral.org/teacher-resources/graph-maker-free-online
Access to free Simulation Modeling Analysis (SMA) Software	Borckardt, J.J. SMA Simulation Modeling Analysis: Time Series Analysis Program for Short Time Series Data Streams. Published 2006. http://www.clinicalresearcher.org/software.htm.

When an established tool is required for systematic review, we recommend use of the What Works Clearinghouse (WWC) Tool because it has well-defined criteria and is developed and supported by leading experts in the SC research field in association with the Institute of Education Sciences.¹⁸ The WWC documentation provides clear standards and procedures to evaluate the quality of SC research; it assesses the internal validity of SC studies, classifying them as “Meeting Standards”, “Meeting Standards with Reservations”, or “Not Meeting Standards”.^1,18 Only studies classified in the first two categories are recommended for further visual analysis. Also, WWC evaluates the evidence of effect, classifying studies into “Strong Evidence of a Causal Relation”, “Moderate Evidence of a Causal Relation”, or “No Evidence of a Causal Relation”. Effect size should only be calculated for studies providing strong or moderate evidence of a causal relation.

The Single-Case Reporting Guideline In BEhavioural Interventions (SCRIBE) 2016 is another useful SC research tool developed recently to improve the quality of single-case designs.⁵⁷ SCRIBE consists of a 26-item checklist that researchers need to address while reporting the results of SC studies. This practical checklist allows for critical evaluation of SC studies during study planning, manuscript preparation, and review.

Summary

Single-case studies can be designed and analyzed in a rigorous manner that allows researchers strength in assessing causal relationships among interventions and outcomes, and in generalizing their results.^2,12 These studies can be strengthened via incorporating replication of findings across multiple study phases, participants, settings, or contexts, and by using randomization of conditions or phase lengths.¹¹ There are a variety of tools that can allow researchers to objectively analyze findings from SC studies.⁵⁶ While a variety of quality assessment tools exist for SC studies, they can be difficult to locate and utilize without experience, and different tools can provide variable results. The WWC quality assessment tool is recommended for those aiming to systematically review SC studies.^1,18

SC studies, like all types of study designs, have a variety of limitations. First, it can be challenging to collect at least five data points in a given study phase. This may be especially true when traveling for data collection is difficult for participants, or during the baseline phase when delaying intervention may not be safe or ethical. Power in SC studies is related to the number of data points gathered for each participant so it is important to avoid having a limited number of data points.^12,58 Second, SC studies are not always designed in a rigorous manner and, thus, may have poor internal validity. This limitation can be overcome by addressing key characteristics that strengthen SC designs (Table 2).^1,14,18 Third, SC studies may have poor generalizability. This limitation can be overcome by including a greater number of participants, or units. Fourth, SC studies may require consultation from expert methodologists and statisticians to ensure proper study design and data analysis, especially to manage issues like autocorrelation and variability of data.² Fifth, while it is recommended to achieve a stable level and rate of performance throughout the baseline, human performance is quite variable and can make this requirement challenging. Finally, the most important validity threat to SC studies is maturation. This challenge must be considered during the design process in order to strengthen SC studies. ^1,2,12,58

SC studies can be particularly useful for rehabilitation research. They allow researchers to closely track and report change at the level of the individual. They may require fewer resources and, thus, can allow for high-quality experimental research, even in clinical settings. Furthermore, they provide a tool for assessing causal relationships in populations and settings where large numbers of participants are not accessible. For all of these reasons, SC studies can serve as an effective method for assessing the impact of interventions.