Abstract
Selecting a quantitative measure to guide decision making in single-case experimental designs (SCEDs) is complicated. Many measures exist and all have been rightly criticized. The two general classes of measure are overlap-based (e.g., percentage nonoverlapping data) and distance-based (e.g., Cohen’s d). We compare several measures from each category for Type I error rate and power across a range of designs using equal numbers of observations (i.e., 3–10) in each phase. Results showed that Tau and the distance-based measures (i.e., RD and g) provided the highest decision accuracies. Other overlap-based measures (e.g., PND, dual-criterion method) did not perform as well. It is recommended that Tau be used to guide decision making about the presence/absence of a treatment effect, and RD or g be used to quantify the magnitude of the treatment effect.
Supplementary Information
The online version contains supplementary material available at 10.1007/s40614-021-00317-8.
Keywords: Statistical analysis, Ratio of distances, Tau, Decision making
Visual analysis has been the dominant approach historically for evaluating treatment efficacy in single-case experimental designs (SCED). The logic of visual analysis is that effective treatments provide obvious evidence of utility, and their effect(s) will be evident in a plot of the data. The main question regarding visual analysis is whether the pattern reflects chance variation only, or chance plus the systematic effect of the treatment. A requirement for effective decision making based on visual analysis, therefore, requires that the analyst have sophisticated knowledge of what can happen simply by chance. Human decision-making research has shown that differentiating random patterns from nonrandom patterns is naturally difficult for humans (e.g., Bar-Hillel & Wagenaar, 1991; Hahn & Warren, 2009; Nickerson, 2002). Many studies (e.g., Voss et al., 2012) have demonstrated the tendency of humans to see patterns in random displays. This natural bias can be exacerbated when there is an expectation or desire for a particular outcome in the data/plot. Indeed, several studies have found visual analysis of SCED data to be unreliable (DeProspero & Cohen, 1979; Fisher et al., 2003; Lanovaz et al., 2020; Ninci et al., 2015; Wolfe et al., 2016, 2018). Due to the problems with visual analysis, many statistical complements to visual analysis have been researched.
Key variables assessed when performing visual analysis of data plots include degree of overlap between baseline and treatment-phase data points, and consideration of the distances between data points, both within and between phases. Within-phase differences provide information regarding variability and stability. Between-phase differences reflect level and pattern changes due to the effect of the treatment. Many statistical complements to visual analysis can be categorized as either overlap-based measures or distance-based measures, corresponding with these variables. It is logical that in the absence of a treatment effect, overlap between points in the baseline and treatment phase will be likely. However, when a treatment effect is present, the treatment observations should systematically differ from (i.e., not overlap with) baseline observations. However, the extent of the overlap (or nonoverlap) will vary depending on the magnitude of the effect, the numbers of observations (and therefore the number of possible outcomes), and other factors (e.g., variance of observations). It is more likely to have 100% nonoverlap between phases if there is a large magnitude effect, small numbers of observations per phase, or more stability in the baseline.
Overlap-based measures include percentage of nonoverlapping data (PND; Scruggs et al., 1987), percentage exceeding the median (PEM; Ma, 2006), Tau-U (Parker et al., 2011), and dual-criterion (DC; Fisher et al., 2003) methods. These measures can inform decisions regarding the presence or absence of an effect, assuming their null distributions are known, but provide limited information regarding the magnitude of the effect (cf. Carter, 2013). The values from these measures will be correlated with the magnitude of the effect because higher percentages of nonoverlap are likely when an effect is present. However, 100% overlap can occur when all treatment values are one point greater than the highest baseline value and when they are 100 points higher. Overlap is still 100% in both cases, but the magnitude is much greater in the latter case; the distance information simply is not part of the statistical computation. Thus, overlap-based statistics can be used to guide decisions about whether a treatment effect is present or not, but should not be relied upon to estimate the magnitude of the treatment effect.
Another concern with application of overlap-based measures is that a standard set of criteria is often used for decision making, rather than tailored criteria. A commonly cited method is to conclude that a treatment is “very effective” if the percentage nonoverlapping data (PND) is 90% or greater, “effective” if 70%–89%, “questionable” if PND is 50%–69%, and “ineffective” if less than 50% (Scruggs & Mastropieri, 1998). However, as noted above and by others (e.g., Allison & Gorman, 1993), PND probabilities vary with numbers of observations. With few observations, any evidence of an effect would require 100% nonoverlap. However, as the number of observations per phase increases, much lower values of PND (e.g., ~50%) could be equally probable evidence of an effect. Thus, overlap-based measures can be very informative, but only when the probabilities of values in the absence of an effect for the particular design structure are known. If these probabilities are known, then statistical decision-making accuracies for particular values can be determined.
Distance-based measures used in SCED are typically variations of Cohen’s d (Cohen, 1988), which is one of the most commonly used measures of effect size in group research. Cohen’s d quantifies group mean differences in standard deviation (SD) units. Many variations exist that were adapted for special cases involving difficulties such as unequal variances, correlated observations, and small sample sizes (see Lakens, 2013, for a review of many of the options). Perhaps the best alternative for SCED is Hedges’s g, which corrects for unequal sample sizes (or numbers of observations per phase in SCED) and for overestimation bias associated with small sample sizes. Others have proposed alternatives designed specifically for SCED that provide the opportunity to correct for slope and/or autocorrelation (Manolov & Solanas, 2008; McKnight et al., 2000).
A recently introduced alternative to Cohen’s d or Hedges’s g for SCED is the ratio of distances (RD; Carlin & Costello, 2018) statistic, which compares between-phase pairwise differences to within-phase pairwise distances. RD is the ratio of absolute between-phase pairwise distances divided by the sum of the within-phase distances. Thus, the structure of the statistic is similar to that of most inferential and effect size statistics. The numerator reflects variance due to sampling error plus an effect, and the denominator is an estimate of sampling error alone. The ratio, therefore, increases as the magnitude of the effect increases. This metric is similar computationally to Cohen’s d, but is more consistent with the logic of SCED visual analysis. As a rule, when viewing a data plot, the distances between points are considered in the context of the variability seen in the baseline phase. RD is designed to reflect these variations within and between phases.
The studies reported below assess distributional properties of several of the statistics used for analyzing data from SCEDs. The studies also compute decision-making accuracies for the statistics to determine those that best guide decision making (i.e., whether variation is likely due to chance vs. a systematic effect). Study 1 assessed distributional properties of RD and PND only, whereas Study 2 expanded the list of statistics considered to include Tau, PEM, DC, and Hedges’s g. The studies provide much information regarding critical values for decision-making, the limitations of using a single set of criteria for evaluating overlap measures across AB design structures (i.e., different numbers of observations in the conditions), and the decision-making accuracies of these statistics.
Study 1: RD and PND Distributions
The purpose of Study 1 was to establish the distributions of RD and PND for a range of AB design structures when no effect is present. Knowing the distribution of values of a statistic when no effect is present allows identification of a critical value for decision making. A critical value is a cut point on a distribution at which the decision to reject or accept the null hypothesis changes. Values greater than the critical value are considered support for rejection of the null hypothesis (i.e., a treatment effect is present), whereas values less than the critical value are considered consistent with the null hypothesis. That is, they occur with high probability when the null hypothesis is true. The conventional critical value is the point on the null distribution beyond which less than 5% of expected values fall.
Because RD is a new distance-based statistic, little is known about its distributional properties across changing numbers of observations per phase. Carlin and Costello (2018) proposed a critical value of RD for statistical decision making of 1.20, but this was based on a single AB design structure. The critical value for deciding whether an effect is present or not should vary with numbers of observations, so more work is needed to identify the best values to use for a broader range of AB structures. This specification will provide much more information regarding error rates and power when using the RD statistic, and form the foundation for Study 2.
PND is an intuitively appealing overlap measure of treatment effect in SCED, which is likely why its use has persisted for decades (see Scruggs & Mastropieri, 2013). However, in its common usage, a single set of guidelines is used to make judgments about the efficacy of a treatment. However, as with RD and most other inferential statistics, the values will vary as numbers of observations change. Study 1, therefore, is designed also to evaluate the probabilities of PND values across multiple AB design structures (i.e., numbers of observations per condition or phase), to inform decision making and Study 2.
Method
Data Generation
Distributions of RD and PND were assessed for AB designs with 3–10 observations per phase. For each number of observations, 5,000 data sets were generated randomly using Microsoft Excel. Values were randomly selected from a distribution with μ = 20 and σ = 4. The formula used in each cell of the random value generator was “=NORMINV(RAND(),20,4).” Ten thousand phases of data were generated. Odd numbered sets were considered baselines and even number sets were considered treatment phases. This process created 5,000 comparisons for which no effect was present, thus establishing the degree of variability expected by chance for each statistic.
Procedure
For each of the 5,000 data sets, RD and PND were computed. A modified version of RD was used in this research. Rather than using the correction procedures proposed in Carlin and Costello (2018), the numerator and denominator were weighted to equate numbers of sums. For example, for a three-observation per phase AB design, there would be nine numerator values summed, but only six values summed in the denominator. This leads to an artificial inflation of the RD statistic. In the current studies, RD values were multiplied by the reciprocal of the numbers of sums in the numerator and denominator. For example, for the three-observation example, RD would be multiplied by 6/9.
The distribution of RD was assessed via basic descriptive statistics (mean, median, standard deviation, and skew). For PND, three measures were computed for each comparison. PND was computed, as were nonoverlap in the negative (i.e., below the lowest baseline value) and positive (i.e., greater than the highest baseline value) directions. This was done so the one-tailed nature of most statistical decisions in SCED could be considered. PND as a global measure could be 100% in different ways (i.e., unidirectional or bidirectional). There could be 100% nonoverlap with some points below the baseline and others above the baseline values (bidirectional), or PND could be 100% in a single direction (i.e., unidirectional). It was assumed for computational purposes that the hypothesis was unidirectional and, therefore, only unidirectionally positive PND values were considered to be findings of an “effect.” This is the standard method for computing PND for SCED.
Critical values of RD, assuming a Type I error rate of 5%, were computed for one-tailed tests for each AB design structure. Critical values were estimated in three ways. The first data-based approach involved identifying the value cutting off the upper 5% (i.e., 250/5,000) of values. Thus, value 250 in the ordered series was considered the critical value. An estimate also was computed based on the absolute RD values. In this case, all RD values were converted to positive values and value 500 in the ordered series was identified as the critical value. Finally, a normal approximation (i.e., 1.645 x SD of RD) was computed for each AB design structure, but considered valid only if the sampling distributions of RD were normal. In the case of normal distributions these three methods for estimating the critical value should be very close, so this was a method for assessing reliability of the value.
Distributions were considered normal if the skew was within the 95% confidence interval of 0 skew. The standard error of skew for N = 5,000 is .035 and the 95% confidence interval (CI) varies from -.068 to +.068. Distributions were considered normal if skew, therefore, was within this range. For distributions with skew outside this range, the normal approximation values were computed, but should be considered carefully.
Results
Descriptive Statistics for RD
Results (see Table 1) showed that the distributions of RD approximated normality with a central tendency of 0. There was slight, positive skew for AB designs with five or fewer observations, but the distributions approached normality with six or more observations per phase. The most extreme skew (i.e., .361) was that for the smallest AB designs with just three observations per phase. This was largely due to one very extreme value of 10.18. When this outlier was removed, the skew was reduced to .149. As numbers of observations increased, the distributions narrowed (i.e., SDs and ranges decreased). Skew was within the 95% range of zero when number of observations was six or more per phase.
Table 1.
Descriptive statistics for RD across designs with varying numbers of observations per phase
| Number of Observations Per Phase | ||||||||
|---|---|---|---|---|---|---|---|---|
| 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | |
| M | -.001 | .016 | .014 | .006 | .004 | -.005 | -.003 | .006 |
| MD | .008 | .013 | .014 | .007 | .012 | -.002 | -.006 | .007 |
| SD | .983 | .736 | .626 | .547 | .511 | .482 | .439 | .420 |
| Min | -6.01 | -4.64 | -3.13 | -2.43 | -2.42 | -2.38 | -1.82 | -1.48 |
| Max | 10.18 | 6.98 | 3.97 | 2.47 | 2.33 | 2.07 | 2.01 | 1.80 |
| Skew | .361 | .167 | .134 | .018 | .036 | -.005 | -.012 | -.011 |
Critical values of RD (see Table 2) decreased as numbers of observations increased, as expected. The three methods for estimating the critical values provided similar results and converged as numbers of observations increased. The most conservative (i.e., highest value) critical values for decision making were the z-based estimates. Data for Tables 1 and 2 are provided in the Supplemental Files.
Table 2.
Critical values (one-tailed) of RD by design structure
| Number of Observations Per Phase | ||||||||
|---|---|---|---|---|---|---|---|---|
| 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | |
| Data | 1.473 | 1.193 | 0.999 | 0.891 | 0.839 | 0.785 | 0.714 | 0.675 |
| ABS | 1.490 | 1.168 | 1.004 | 0.885 | 0.826 | 0.795 | 0.713 | 0.681 |
| z | 1.616 | 1.211 | 1.029 | 0.900 | 0.840 | 0.793 | 0.722 | 0.690 |
Descriptive Statistics for PND
Table 3 shows the percentage of data sets in each condition with a particular number of nonoverlapping data points. For example, for three observations per phase, 20.5% of data sets had full overlap (i.e., zero nonoverlapping data values) and 19.74% had nonoverlap in both directions (i.e., mix). These occurred most frequently, as expected, because there was no true effect present. At the extremes, 5.44% of the data sets had 100% nonoverlap (i.e., three points) in the negative direction and 5.40% of data sets had 100% nonoverlap in the positive direction. The most common outcomes across all observation numbers were values of 0 and situations with nonoverlap in both directions (i.e., “mix”). The values also were symmetrical around 0, as variation is as likely in the negative as positive direction when no true effect is present.
Table 3.
Percentage data sets, for each design structure, with particular numbers of nonoverlapping data points
| Number of Observations per Phase | ||||||||
|---|---|---|---|---|---|---|---|---|
| # ND* | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
| -10 | 0.00 | |||||||
| -9 | 0.00 | 0.00 | ||||||
| -8 | 0.00 | 0.00 | 0.02 | |||||
| -7 | 0.04 | 0.02 | 0.16 | 0.16 | ||||
| -6 | 0.14 | 0.20 | 0.24 | 0.20 | 0.60 | |||
| -5 | 0.26 | 0.32 | 0.42 | 0.90 | 0.74 | 0.80 | ||
| -4 | 1.16 | 1.60 | 1.60 | 1.78 | 1.68 | 1.26 | 2.10 | |
| -3 | 5.44 | 4.40 | 3.68 | 3.52 | 3.76 | 3.68 | 3.38 | 3.68 |
| -2 | 9.48 | 7.76 | 7.86 | 7.48 | 7.04 | 7.48 | 6.98 | 6.70 |
| -1 | 15.18 | 14.38 | 13.64 | 13.22 | 13.20 | 12.72 | 13.54 | 13.48 |
| 0 | 20.50 | 22.46 | 21.76 | 23.06 | 23.32 | 22.90 | 22.98 | 22.38 |
| Mix | 19.74 | 20.84 | 22.70 | 23.40 | 23.22 | 23.78 | 23.40 | 24.60 |
| 1+ | 14.72 | 14.66 | 14.12 | 12.88 | 13.58 | 12.70 | 13.48 | 12.84 |
| 2+ | 9.54 | 8.90 | 8.48 | 8.08 | 7.16 | 7.28 | 6.96 | 6.62 |
| 3+ | 5.40 | 4.04 | 3.76 | 4.02 | 3.92 | 3.92 | 3.60 | 3.12 |
| 4+ | 1.40 | 1.66 | 1.70 | 1.44 | 1.62 | 1.98 | 1.78 | |
| 5+ | 0.48 | 0.44 | 0.54 | 0.72 | 1.00 | 0.68 | ||
| 6+ | 0.14 | 0.34 | 0.24 | 0.30 | 0.34 | |||
| 7+ | 0.04 | 0.12 | 0.04 | 0.10 | ||||
| 8+ | 0.00 | 0.00 | 0.00 | |||||
| 9+ | 0.00 | 0.00 | ||||||
| 10+ | 0.00 |
Note. # ND is number of nonoverlapping data points. Negative numbers represent numbers of nonoverlapping values below the baseline minimum and positive values represent numbers of nonoverlapping values above the baseline maximum.
Table 3 provides the critical information necessary for specifying probabilities of outcomes for statistical decision making using PND. Data for these distributions are provided in the Supplemental Files. A common statistic used for statistical decision making is the p-value. This is the probability of obtaining the observed result (or something more extreme in the predicted direction) by chance. The standard criterion used is a p-value of .05 (or 5%). This is why the critical value of a statistic is the value beyond which 5% or less of the distribution falls, when no effect is present. The Table 3 values in bold designate the values corresponding to a p-value of .05 or less. Figure 1 shows the same cumulative percentages of data sets having a specified number of nonoverlapping points in the predicted direction for each level of number of observations. The critical value for statistical decision making would be the value (i.e., number of nonoverlapping points) at which the cumulative percentage drops below 5%. As seen in Table 3 and the Fig. 1, this value is four for all conditions. Thus, if using PND to make decisions regarding the presence of an effect, the critical number of nonoverlapping points in the predicted direction is four, with no nonoverlapping values in the opposite direction. That is, the nonoverlap must be unidirectional to conclude a treatment effect is present.
Fig. 1.
Cumulative percentages of data sets with particular numbers of non-overlapping data points for each of the AB design structures
Discussion
Knowledge of the null distributions of RD and PND informs decision making using these statistics. Both statistics have predictable and consistent distributions, which supports their usage for statistical decision making. Further, the current data qualify published criteria for usage of these statistics. This enhanced knowledge of the distributions of these statistics when effects are not present should greatly improve decision making validity.
Carlin and Costello (2018) suggested that a critical value of 1.20 be used for RD in general. However, this was based on limited knowledge of the variability of RD as the number of observations varies. As expected, the current data show that the critical value of RD decreases as the number of observations per phase increases. Therefore, statistical decision validity will be improved if the critical value is matched to the particular number of observations per phase. Using the z-based critical values in Table 2 will improve decision making by increasing statistical power (i.e., the probability of detecting a true effect, when present) and reducing the inflated Type II error rate associated with using a critical value of 1.20 for all numbers of observations. The z-based critical values are those we would recommend because they were the most conservative in all but one instance (i.e., eight observations). Therefore, they should provide the most power while keeping the Type I error probability below 5%.
The distributions of PND also were informative regarding commonly used criteria for decision making. The oft-cited guidelines of Scruggs and Mastropieri (1998) are that 90% PND or greater indicates a strong treatment effect, 70%–89% PND is consistent with an effect, 50%–69% indicates a marginal effect, and less than 50% PND supports a finding of no effect of treatment. However, as with RD above, use of the same decision criteria for AB designs with varying numbers of observations per phase is not justified, and will greatly reduce decision accuracy. For example, with three observations per phase, results showed that no value of PND occurred on less than 5% of data sets. Thus, there is no critical value that would keep the Type I error rate below 5%. The probability of having 100% nonoverlap simply by chance is too high. Thus, 90% or greater is not a valid decision criterion with so few data points.
The critical percentages nonoverlap for AB designs with more than three observations per phase also did not support the general guidelines of Scruggs and Mastropieri (1998). In fact, the data of Study 1 (e.g., Fig. 1) showed that percentage nonoverlap likely is not the best criterion to consider. Rather, the number of nonoverlapping points is the better criterion for decision making. For AB designs with 4–10 observations per phase, the critical value was four nonoverlapping points in the predicted direction, with no opposite direction nonoverlap. Using this frequency-based critical value resulted in a Type I error rate below 5% for all of these observation numbers. This should greatly simplify statistical decision making using this statistic. If using percentages, the percentage would change with each additional observation per phase. For example, with four observations per phase, 100% (4/4) nonoverlap would be needed to conclude a treatment is present, but just 40% nonoverlap (4/10) would be needed to reach the same conclusion with 10 observations per phase. The decreasing percentage nonoverlap necessary to reject the null hypothesis reflects the increasing power of the statistical tests as observations are added to the AB design.
Study 2: Decision-Making Accuracies
The purpose of Study 2 was to replicate the findings of Study 1 using distributions with different parameters (μ = 50, σ = 10), and to expand the set of statistics compared. The statistics compared were considered to be of two types: (1) overlap-based measures used to guide decisions regarding the presence/absence of an effect, and (2) distance-based statistics used to estimate the magnitude of the effect (i.e., effect size). The overlap-based measures were PND, the dual-criterion (DC) method, percentage exceeding the median (PEM), and Tau-based z. The effect size measures considered were RD and Hedges’s g. These statistics were assessed across three design structures: (1) 3 observations per phase, (2) 5 observations per phase, and (3) 10 observations per phase. For each condition, effect sizes of d = 0, d = 1, and d = 2 were studied. Thus, data sets with and without effects present were assessed in Study 2.
DC is a method developed by Fisher et al. (2003) in which the percentage of treatment data points above both of two criteria are counted and compared to a specified percentage. This method was proposed as an alternative to PND and has been growing in popularity. The two criteria in the DC method are percentage of treatment values above the baseline mean and percentage of treatment values above the regression line, based on the baseline observations. The statistic computed is percentage of treatment-phase data points above both criteria. The DC method has not been recommended for AB designs with fewer than five observations per phase because no degree of overlap would be considered significant with a Type I error rate of .05. This is similar to the finding with PND in Study 1 when just three observations per phase were used. We decided to assess DC even for three observations per phase in this study to provide a comparison to the other statistics and the AB designs with more observations. For decision making, the percentage above both criteria considered significant was 100% for the 3- and 5-observations per phase AB designs, and 80% for the 10-observations AB designs. These critical values are those cited by Fisher et al.
An alternative measure proposed by Fisher et al. (2003) was the conservative dual-criterion (CDC) method. This was not considered in this study due to its expected low power. Based on the work of Fisher et al., both the DC and CDC methods seemed to have low power, so we opted to evaluate the more powerful of the two.
PEM (Ma, 2006) is similar to the first criterion in the DC method but uses the median rather than the mean. PEM was developed to address several concerns with PND, most notably low power and susceptibility to floor and/or ceiling effects in baseline. One potential disadvantage of PEM is that the expected value, in the absence of an effect, is 50%. Thus, the range for detecting an effect is relatively limited. PND, on the other hand, has an expected value of 0, when an effect is not present. Further, because the scales of PND and PEM are different, the criteria for decision making will differ. However, there is no generally agreed upon system for interpreting PEM as there is for PND (e.g., Scruggs & Mastropieri, 1998). Data from this study should provide some guidance regarding critical values of PEM.
Tau is an overlap-based measure that uses the z-statistic and one-tailed testing for decision making (e.g., Parker et al., 2011). An advantage of Tau-based z relative to other overlap-based measures is the greater number of possible values for a particular data set. This makes it a more powerful statistic. In addition, because Tau uses the z statistic, the critical value will always be 1.645, given a one-tailed hypothesis test.
Hedges’s g is not commonly used to make decisions regarding the simple presence or absence of effect. This statistic is used to quantify the magnitude of the effect, if one is established. Tables of critical values for g, therefore, are not available. However, we believe some knowledge of the distributions would be useful for making the most informed decisions possible, as was established for RD in Study 1. Beyond this, we also wanted to assess the relationship between RD and g values in terms of absolute value and correlation of values. We used Hedges’s g variation outlined by Lakens (2013) for a pooled error term and the correction for small sample size.
Method
Five thousand data sets were generated for each AB design structure (3, 5, and 10 observations per phase) and for each of three effect sizes, d = 0, d = 1, and d = 2, using the same Excel procedures as Study 1. For d = 0, all values were randomly selected from a distribution with μ = 50 and σ = 10. For d = 1, baseline values were selected from a distribution with μ = 50 and σ = 10, and treatment phase data were generated from a distribution with μ = 60 and σ = 10. For d = 2, treatment data were generated from a distribution with μ = 70 and σ = 10. For the 45,000 comparisons, RD, PND, PEM, DC, Tau z, and g were computed.
Decision-making statistics computed included the positive predictive value (PPV) and negative predictive value (NPV). PPV provides an estimate of the proportion of identified effects that are true effects as opposed to Type I errors. NPV provides the same value for findings of no effect (i.e., correct rejections vs. misses). In a standard experiment with power = .80 and a Type I error rate of .05, assuming equal numbers of effects and noneffects, PPV is .9412 and NPV is .8261. The percentage correct decisions would be .875. These were used as comparisons for the values obtained for the measures studied.
Results
Effect Size Statistics: RD and g
Descriptive statistics for RD and g are shown in Tables 4, 5 and 6. When no effect was present (Table 4), both measures centered around 0 and variability decreased substantially as the numbers of observations increased. With just three observations per phase the ranges for RD and g were extreme. With effects of size d =1 and d = 2 present (Tables 5 and 6), mean g values were very close to the d values, with medians slightly lower. RD provided slightly lower estimates of effect sizes than did g for most conditions. RD was slightly less variable than g for 5- and 10-observations conditions. Correlations between RD and g were greater than .99 for all conditions and effect sizes. Overall, results showed that RD and g provide very similar information regarding effect sizes.
Table 4.
RD and g descriptive statistics for d = 0
| RD | g | |||||
|---|---|---|---|---|---|---|
| 3 | 5 | 10 | 3 | 5 | 10 | |
| Mean | .0085 | .0131 | .0092 | .0079 | .0136 | .0099 |
| Median | -.0048 | .0187 | .0088 | -.0044 | .0196 | .0093 |
| SD | .9430 | .6418 | .4146 | .9020 | .6726 | .4539 |
| Skew | -.1301 | -.0046 | .0170 | -.0896 | .0007 | .0156 |
| MIN | -10.80 | -3.18 | -1.73 | -10.13 | -3.21 | -1.91 |
| MAX | 8.58 | 3.98 | 1.61 | 8.37 | 4.28 | 1.74 |
Table 5.
RD and g descriptive statistics for each design structure with d = 1
| RD | g | |||||
|---|---|---|---|---|---|---|
| 3 | 5 | 10 | 3 | 5 | 10 | |
| Mean | 1.031 | 0.954 | 0.915 | 0.985 | 1.001 | 1.002 |
| Median | 0.872 | 0.903 | 0.887 | 0.844 | 0.948 | 0.972 |
| SD | 1.062 | 0.697 | 0.447 | 1.006 | 0.729 | 0.489 |
| Skew | 1.833 | 0.752 | 0.400 | 1.661 | 0.761 | 0.394 |
| MIN | -3.01 | -1.17 | -0.60 | -3.07 | -1.29 | -0.63 |
| MAX | 12.62 | 5.44 | 3.74 | 10.03 | 5.87 | 4.11 |
Table 6.
RD and g descriptive statistics for each design structure with d = 2
| RD | g | |||||
|---|---|---|---|---|---|---|
| 3 | 5 | 10 | 3 | 5 | 10 | |
| Mean | 2.096 | 1.905 | 1.835 | 2.003 | 1.999 | 2.010 |
| Median | 1.791 | 1.794 | 1.788 | 1.712 | 1.885 | 1.958 |
| SD | 1.427 | 0.845 | 0.544 | 1.355 | 0.880 | 0.593 |
| Skew | 2.561 | 1.406 | 0.705 | 2.581 | 1.418 | 0.702 |
| MIN | -1.19 | -0.22 | 0.33 | -1.22 | -0.22 | 0.36 |
| MAX | 21.73 | 10.75 | 5.26 | 22.14 | 11.35 | 5.83 |
Overlap-Based Measures (d = 0)
The distributions of PND (Table 7) replicated those from Study 1. For three observations, the 100% nonoverlap value resulted in a Type I error rate just greater than 5%. For 5-observation AB designs, the critical value was 80% nonoverlap (i.e., four nonoverlapping data points in the predicted direction), and for 10 observations the critical value was 40% nonoverlap (i.e., four nonoverlapping points of 10). These critical values matched those from Study 1.
Table 7.
Percentage data sets, for each design structure, with particular numbers of nonoverlapping data Points (d = 0)
| # ND | 3 | 5 | 10 | ||||||
|---|---|---|---|---|---|---|---|---|---|
| PND | DC | PEM | PND | DC | PEM | PND | DC | PEM | |
| 0 | 19.34 | 41.72 | 20.06 | 21.74 | 26.68 | 7.66 | 22.62 | 9.28 | 0.68 |
| 1 | 15.54 | 27.14 | 30.12 | 13.70 | 21.20 | 17.82 | 11.94 | 10.96 | 3.30 |
| 2 | 10.20 | 20.48 | 29.36 | 8.30 | 19.32 | 23.56 | 5.96 | 12.20 | 7.18 |
| 3 | 5.06 | 10.66 | 20.46 | 4.40 | 17.70 | 23.96 | 3.48 | 13.98 | 12.32 |
| 4 | 1.90 | 10.96 | 18.62 | 1.94 | 13.88 | 17.04 | |||
| 5 | 0.38 | 4.14 | 8.38 | 0.76 | 14.14 | 17.62 | |||
| 6 | 0.40 | 11.44 | 16.84 | ||||||
| 7 | 0.12 | 7.90 | 12.80 | ||||||
| 8 | 0.00 | 4.30 | 7.80 | ||||||
| 9 | 0.00 | 1.66 | 3.70 | ||||||
| 10 | 0.00 | 0.26 | 0.72 | ||||||
The distributions of DC (Table 7) supported the work of Fisher et al. (2003). For three observations, there was no value that met the criterion of a Type I error rate of 5% or less. The 100% nonoverlap value (i.e., three nonoverlapping points) occurred more than 10% of the time. For AB designs with five observations per phase, the Type I error rate was 4.14% for the 100% criterion (i.e., five nonoverlapping points). The Type I error rate for the 80% criterion for AB designs with 10 observations per phase was 6.22% (i.e., 4.30 + 1.66 + 0.26). These Type I error rates replicated those of Fisher et al. If one desired a critical value for 10 observations that kept the Type I error rate below 5%, then the 90% criterion would have a Type I error rate of 1.92% (i.e., 1.66 + 0.26), but substantially less power.
Distributions for PEM (Table 7) showed that no values resulted in Type I error rates of 5% or less for the three- and five-observations per phase AB designs. The error rates were 20.46% for three-observations AB designs and 8.38% for five observations AB designs. For the 10-observations per phase AB designs, a PEM value of 90% or greater resulted in a Type I error rate of 4.42%.
Critical Values for Statistical Decision Making
Critical values of RD identified in Study 1 generalized well to these data sets. For three observations (Table 8), the absolute estimate of 1.49 was used due to the skew associated with small numbers of observations. For the 5- and 10-observation conditions (Tables 9 and 10, respectively), the normal distribution estimates (1.029 and 0.69) were used. Critical values for g were data-based, with the value defining the upper 5% of each distribution used. These values were 1.423 for the 3-observations condition, 1.0874 for the 5-observations condition, and 0.747 for the 10-observations condition.
Table 8.
Decision making probabilities for statistics (3 observations per phase)
| Statistic CV* | Statistic (criterion) | |||||
|---|---|---|---|---|---|---|
| RD 1.490 | g 1.423 | Tau-U z 1.645 | PND 100% | DC 100% | PEM 100% | |
| Type I Error | .0510 | .0500 | .0506 | .0506 | .1066 | .2046 |
| Power d = 1 | .2526 | .2560 | .2628 | .2628 | .3628 | .5560 |
| PPV d = 1 | .8320 | .8366 | .8385 | .8385 | .7729 | .7310 |
| NPV d = 1 | .5594 | .5608 | .5629 | .5629 | .5837 | .6418 |
| P(correct) | .6008 | .6030 | .6061 | .6061 | .6281 | .6757 |
| Power d = 2 | .6262 | .6278 | .6186 | .6186 | .6274 | .8714 |
| PPV d = 2 | .9247 | .9262 | .9244 | .9244 | .8548 | .8099 |
| NPV d = 2 | .7174 | .7185 | .7134 | .7134 | .7057 | .8608 |
| P(correct) | .7876 | .7889 | .7840 | .7840 | .7604 | .8334 |
*CV critical value
Table 9.
Decision making probabilities for statistics (5 observations per phase)
| Statistic CV | Statistic (criterion) | |||||
|---|---|---|---|---|---|---|
| RD 1.029 | g 1.0874 | Tau-U z 1.645 | PND 80% | DC 100% | PEM 100% | |
| Type I Error | .0510 | .0500 | .0526 | .0228 | .0414 | .0838 |
| Power d = 1 | .4198 | .4108 | .4010 | .2204 | .2736 | .8106 |
| PPV d = 1 | .8917 | .8915 | .8840 | .9063 | .8686 | .9063 |
| NPV d = 1 | .6206 | .6172 | .6126 | .5562 | .5689 | .8287 |
| P(correct) | .6844 | .6804 | .6742 | .5988 | .6161 | .8634 |
| Power d = 2 | .8866 | .8870 | .8710 | .6624 | .6202 | .9626 |
| PPV d = 2 | .9456 | .9466 | .9430 | .9667 | .9374 | .9199 |
| NPV d = 2 | .8933 | .8937 | .8802 | .7432 | .7162 | .9608 |
| P(correct) | .9178 | .9185 | .9092 | .8198 | .7894 | .9394 |
Table 10.
Decision making probabilities for statistics (10 observations per phase)
| Statistic CV | Statistic (criterion) | |||||
|---|---|---|---|---|---|---|
| RD 0.69 | g 0.747 | Tau-U z 1.645 | PND 40% | DC 80% | PEM 90% | |
| Type I Error | .0476 | .0502 | .0550 | .0322 | .0622 | .0442 |
| Power d = 1 | .6910 | .6968 | .6798 | .3700 | .5140 | .5166 |
| PPV d = 1 | .9356 | .9328 | .9251 | .9199 | .8921 | .9212 |
| NPV d = 1 | .7550 | .7580 | .7469 | .6057 | .6587 | .6641 |
| P(correct) | .8217 | .8233 | .8124 | .6689 | .7259 | .7362 |
| Power d = 2 | .9952 | .9956 | .9936 | .8736 | .8584 | .9492 |
| PPV d = 2 | .9544 | .9520 | .9475 | .9645 | .9324 | .9555 |
| NPV d = 2 | .9950 | .9954 | .9933 | .8845 | .8688 | .9495 |
| P(correct) | .9738 | .9727 | .9693 | .9207 | .8981 | .9525 |
The critical value for Tau-z was 1.645 for all conditions. For PND, 100% was the criterion for the 3-observations condition, 80% for the 5-observations conditions, and 40% for the 10-observations condition. The critical value for DC and PEM was 100% for the three- and five-observation conditions. For the 10-observations AB designs, the DC criterion was 80% and that for PEM was 90%. The PEM critical values were 100% for the smaller two designs because no values returned a Type I error rate below 5%. For the 10-observations designs, 90% was the lowest value that maintained a Type I error rate below 5%.
Decision-Making Statistics: Three Observations per Phase
For three-observations AB designs (Table 8), Tau-z and PND were functionally the same statistic. This was due to the fact that both resulted in a significant effect only with the most extreme score (PND = 100%, z = 1.945), and these values were perfectly correlated. Thus, all values match for these statistics.
Powers and decision-making statistics for RD and g also were very similar to each other, and to PND and Tau-z. All measures had very low power for detecting effects of size d = 1. Power was below the typically recommended level (i.e., .80) even for the larger effect size (d = 2). NPV values were significantly lower than PPV values, indicating that Type II errors are of particular concern due to the low power. Three observations per phase are not enough to make reliable decisions regarding the presence/absence of an effect, even for quite large effects.
Despite the elevated Type I error rate associated with DC, it provided little advantage over RD, g, Tau-z, or PND for decision-making accuracy. It provided more power for the smaller effect size, but was associated with lower PPVs for both effect sizes. PEM provided the most power, and therefore had the highest NPVs, and overall accuracies, of all the statistics. However, the extremely high Type I error rate limits its utility for experimental use.
Decision-Making Statistics: Five Observations per Phase
For five-observations AB designs (see Table 9), RD, g, and Tau-z provided clear advantages over PND, DC, and PEM. RD, g, and Tau-z had similar outcomes, with excellent decision-making outcomes for d = 2, but low powers, NPVs, and percentages correct for d = 1. PND and DC had very low power, primarily due to the necessity of using a critical value associated with a lower Type I error rate. PEM again had an elevated Type I error rate, though less so than in the three-observations condition. Overall decision-making outcomes for PEM were better than all other measures, otherwise. However, if RD, g, and Tau-z used critical values with the higher Type I error rate, their statistics likely would be better than those for PEM.
Decision-Making Statistics: 10 Observations per Phase
For 10-observations AB designs (see Table 10), RD, g, and Tau-z again provided the best outcomes. Standards for comparison are PPV = .9412, NPV = .82, and .875 proportion correct. These are the values associated with the conventional design having .80 power and a Type I error rate of .05. associated with the conventions of .80 power and .05 Type I error rate. For d = 1, powers were approximately .69, with PPVs of .93, NPVs of .75, and proportions correct of approximately .82. For the larger effect size (d = 2), values exceeded the standards significantly. PND and DC performed significantly worse than RD, g, and Tau-z, though values were still better than the standard for d = 2. PEM led to better decision outcomes than PND and DC, but performed below the levels of RD, g, and Tau-z.
Data associated with the decision-making statistics for 3-, 5-, and 10-observations AB design structures are in the Supplemental Files.
Discussion
The data from Study 2 provided much information regarding the distributional properties and decision-making accuracies of several statistics used for SCEDs. Regarding the overlap-based measures, Tau-z seemed clearly to be the best statistic for identifying the presence of a true effect. Tau-z’s advantages included the most consistent control of Type I error, a consistent critical value for all AB designs, and the best decision accuracy and highest power when Type I error was controlled at approximately .05. The Type I error rates for Tau-z varied from .0506 to .055, which are slightly elevated, but reasonable. PND had Type I error probabilities significantly below .05 in two conditions, which led to relatively low power. DC had a Type I error rate over .10 for 3-observations AB designs, and over .06 for 10-observations designs. PEM had very elevated Type I error rates of .20 and .08 for the three- and five-observations conditions, respectively.
The critical value for Tau-z is 1.645 for all AB designs, assuming a one-tailed test of significance and Type I error rate of .05. The critical values for PND, DC, and PEM vary by AB design structure and have not been well-defined for most AB designs. The critical value for PND was 100% for 3-observations designs, 80% for 5-observations designs, and 40% for 10-observations designs. These data did not support the common practice of using a single set of criteria for all designs. For DC, there was no critical value for the smallest designs because even 100% nonoverlap occurs with relatively high frequency when no effect is present. For PEM, the same limitation applied to both the three- and five-observations designs. Thus, use of these statistics is limited to designs with larger numbers of observations. For five-observations AB designs, for which Tau-z and DC had somewhat similar Type I error rates, Tau-z provided significantly more power and higher decision accuracies.
For the smaller effect size (i.e., d = 1), no statistics had powers at the recommended level of .80. With three observations per phase, powers were very low, with the highest being .556 for PEM. However, this power came with the cost of a Type I error rate exceeding .20. Powers associated with Tau-z and PND, which controlled the Type I error rate at .05, were just .26. For the five-observations per phase structure, PEM again had the most power (i.e., .81), but also a higher Type I error rate (i.e., .084). Tau-z provided .40 power with a Type I error rate of .053. PND and DC had Type I error rates below .05, but powers of just .22 and .27. For 10-observations designs, Tau-z provided the most power with a Type I error rate of .055. DC had less power and a higher Type I error rate, and PND and PEM had less power and lower Type I error rates. Overall, these AB design structures and statistics were not able to detect effects of size d = 1 at high rates. Detection of significant treatment effects of this magnitude, at recommended powers of .80, requires more than 10 observations per phase, if using these statistics.
For the larger effects, d = 2, Tau-z provided good power while maintaining Type I error rates close to .05. With 3 observations per phase, Tau-z power was .62, and that improved to .87 with 5 observations per phase, and .99 with 10 observations per phase. PND matched Tau-z with three observations, but had substantially less power and overall decision accuracy than did Tau-z for the other AB designs. DC also was similar to Tau-z with three-observations structures, but did not perform as well as Tau-z for the two larger AB designs. PEM had the most power and highest overall percentage correct decisions for the three and five-observations designs, but again largely due to the elevated Type I error rates. The high Type I error rate led to high NPVs, and the high overall accuracy. Though an argument could be made to use PEM, assuming the goal is to maximize overall decision accuracy, using such elevated Type I error rates is not generally accepted in the field. In the 10-observations condition, with PEM having a Type I error rate below .05, power and overall decision accuracy were lower than those of Tau-z.
With regard to the distance-based measures RD and g, there was little to differentiate these statistics. These statistics had similar properties across all conditions, and were correlated .990 or greater for all conditions. Hedges’s g provided the most accurate estimates of the effect sizes defined by d, but this is due to the fact that g is a direct estimator of d. RD returned slightly lower values on average, but again values that were nearly perfectly correlated with g.
RD and g were designed as effect size metrics and do not have a single critical value across all design types, so their usage for this purpose would be complicated. Tau-z, however, is not a measure of effect size, so should not be used for that purpose. Combining Tau-z with one of the two effect size indices seems to be the best option for SCED analysis. Tau-z is designed to guide decisions about the presence of a treatment effect, but not to estimate its magnitude. RD and g were designed as measures of effect size and not primarily to guide decisions about presence of an effect. Thus, pairing the two types of statistics seems the best approach, which should greatly improve decision making accuracy for SCED.
General Discussion
Visual data analysis in SCEDs has been shown to be unreliable, leading to a need for statistical complements. Several SCED statistical measures have been proposed and are popular in the literature. These measures are designed to assist analysts in differentiating chance variation from systematic treatment effects. In the present studies, we used Monte Carlo methods to examine accuracy of several statistical measures. Data sets were generated to compare power and Type 1 error rates for multiple measures (RD, PND, DC, PEM, Tau-based z, and g) across AB designs with 3, 5, and 10 observations per phase, with treatment effect sizes of d = 0, d = 1, and d = 2.
In the aggregate, Tau-z provided the best combination of ease of use and decision-making accuracy for determining the presence of a true treatment effect. The use of a single critical value, regardless of design structure, allows broad, knowledgeable application of the statistic. For other statistics (e.g., RD, g, PND), substantial work would be needed to establish the critical value for design variations. Without that work being done, decision accuracies would be unknown. Further, there were no circumstances under which DC or PEM outperformed the decision-making properties of Tau-z or PND, unless Type I error rates were significantly greater than .05. With three observations per phase, DC and PEM had higher power than the other statistics, but only because the Type I error rates were extremely inflated (10.66% and 20.46%). With five observations per phase, DC had a Type I error rate of just 4.14%, but had low power relative to Tau-z. PEM still had an unacceptably high Type I error rate of 8.38%. With 10 observations per phase, DC had a higher Type I error rate than Tau-z and lower power. PEM had a Type I error rate below 5% but significantly less power than Tau-z. The data analyzed in these studies is available in the Supplemental Files.
Taken together, results show a possible path forward for SCED analysis. The Tau-based z-test should be used to make decisions regarding the presence or absence of a treatment effect, then RD or g could be used to estimate the magnitude of the effect. Our preference is RD because the logic of the statistic is more consistent with visual analysis, and RD and Tau-z are linked computationally. RD is based on the distances between points within and between phases. These are visible in plots of data, and considered in typical visual analysis. RD provides a single value to quantify the visual analysis. Further, RD and Tau-z are computed from the same matrices. Tau is based on signs of pairwise differences between points, whereas RD is computed from the magnitudes of those same pairwise differences. Thus, it seems more natural to pair RD with Tau-z as a system of statistical analysis for SCED that closely aligns with the history of visual analysis in behavior science.
A potential limitation of using RD as a measure of effect size is that it does not generalize as well to group research. This is the main argument for use of a statistic such as g, which is comparable to the widely used Cohen’s d. However, the results demonstrated that RD and g have very similar distributions (e.g., means, SDs, and skews) and correlate very highly (~.99). These results show that despite the fact that RD has a slightly different underlying logic (i.e., ratio of distances between to within phases), it provides a metric scaled similarly to g and d, which estimate mean differences in standard deviation units. The similarity of the scaling of the statistics should promote and allow for cross-disciplinary comparisons and similar interpretations of observed values. The supplemental calculator file allows for easy computation of RD and g for data sets, so they can be directly compared in practice.
Results did not support use of general guidelines for making conclusions based on percentage nonoverlap. Scruggs and Mastropieri (1998), for example, proposed that 90% or higher nonoverlap indicates a strong effect, 70%–90% indicates an effect, 50%–70% a marginal effect, and less than 50% no effect. The data from the present studies showed that values of nonoverlap consistent and inconsistent with an effect varied by AB design structure. Designs with few observations required quite high values of nonoverlap to support a finding of a treatment effect. Relatively low values of percentage nonoverlap (e.g., 40%) were indicative of a treatment effect for designs with more observations. Nonoverlap percentages also provide limited information about the magnitude of the treatment, so other measures (e.g., RD or g) should be used to support any statements associated with treatment effect magnitude.
Results provide some guidance regarding SCED and numbers of observations per phase. With just three observations per phase, no statistics met standard guidelines for decision making accuracy. With the standards of α = .05 and power = .80, decision-making statistics are PPV = .94, NPV = .82, and percentage correct decisions = 87.5. Even with the larger effect size (i.e., d = 2), no statistics met these standards. With five observations per phase, these standards were met for the larger effect size, but not for the smaller effect size (i.e., d = 1). With 10 observations per phase, all statistics attained these standards for the larger effect size. However, for the smaller effect size, all statistics underperformed these standards, though they were much closer than with just five observations per phase. Of the nonoverlap-based statistics, Tau-z was clearly the best, in terms of power, NPV, and percentage correct decisions, for the smaller effect size. Thus, using these statistics for decision making requires at least five observations per phase to detect effects of approximately two SDs (i.e., d = 2), and more than 10 observations to detect smaller effects.
Though the data provide much guidance regarding selection of statistics for SCED, there are some limitations to consider. Most important, all analyses involved comparisons with equal numbers of observations per phase and hypothetical data. These methods were selected because they provided the best information for the specific purpose of this work. Generalization to designs with unequal numbers of observations and with actual data sets still must be established. We have done extensive work, as yet unpublished, to assess these limitations and the results thus far are favorable. However, much work still needs to be done to determine the relationship between RD, g, and d, with field data that likely will be less systematic than that assessed in this research. Additional areas of study will include the impact of slope and autocorrelation on RD and how to correct for slope, if necessary.
The results of this study are consistent with previous research on performance of SCED effect sizes under different design conditions, and lead to several areas of future research. Pustejovsky (2019) also found several overlap measures (including PND and measures not included in the present manuscript) to be very sensitive to number of data points in baseline and treatment phases, and recommended against their continued use. Tau-z, PEM, and a bias-corrected SMD performed well with different numbers of data points per phase, but were found to be affected by other aspects of design. RD was not examined in that study, but one may logically assume some similarity.
Future research should compare decision making with Tau-z and RD (or g) to additional statistics not considered here (e.g., Mann-Whitney U test) and to visual inspection and analysis (see Wolery et al., 2010, for a comparison with overlap measures), because this is the most widely used method of analysis of SCED data. Future research should also examine Tau-z and RD with valid behavioral data, and/or use of data models that correspond with real behavioral data (see Pustejovsky, 2019, for an example). Pustejovsky (2019) recommended the use of bias-corrected log-response-ratio (LRR), because it was the most unaffected by design aspects. LRR has some logical similarities with RD. If Tau-z and RD bear out, future research could compare LRR with Tau-z and RD.
As a final note, statistical testing for SCEDs sometimes has met with resistance on the grounds that the testing is unnecessary, and that the findings from averaged results are antithetical to the philosophy of the designs (many of these critiques focus on significance testing; e.g., Branch, 1999, 2014). Skinner (1938) argued against the use of statistics in favor of experimental control with clearly replicable methods and results. Despite these philosophical and historical pressures against the use of statistics for SCEDs, significant attention has been given to the development of statistics to guide quantification of SCED results, in particular with the proliferation of applied behavioral research where effects may not be as visually clear as in laboratory data. In the experimental analysis of behavior, environmental controlling variables are isolated so that sources of variation are minimized, though not eliminated, but this control is more difficult to achieve in applied and clinical behavior analysis. Thus, although the statistical tools are necessary, so is a strong understanding of the nature of the behavior studied and analyzed.
Supplementary Information
(XLSX 3655 kb)
(XLSX 28 kb)
(XLSX 3655 kb)
(XLSX 603 kb)
(XLSX 1393 kb)
Acknowledgments
Availability of Data
Upon request
Code Availability
Not applicable
Declarations
Conflicts of Interest
None
Ethics Approval
Not applicable
Footnotes
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- Allison DB, Gorman BS. Calculating effect sizes for meta-analysis: The case of the single case. Behaviour Research & Therapy. 1993;31:621–631. doi: 10.1016/0005-7967(93)90115-B. [DOI] [PubMed] [Google Scholar]
- Bar-Hillel M, Wagenaar WA. The perception of randomness. Advances in Applied Mathematics. 1991;12:428–454. doi: 10.1016/0196-8858(91)90029-I. [DOI] [Google Scholar]
- Branch MN. Statistical inference in behavior analysis: Some things significance testing does and does not do. The Behavior Analyst. 1999;22:87–92. doi: 10.1007/BF03391984. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Branch M. Malignant side effects of null-hypothesis significance testing. Theory & Psychology. 2014;24:256–277. doi: 10.1177/0959354314525282. [DOI] [Google Scholar]
- Carlin MT, Costello MS. Development of a distance-based effect size metric for single-case research: Ratio of distances. Behavior Therapy. 2018;49:981–994. doi: 10.1016/j.beth.2018.02.005. [DOI] [PubMed] [Google Scholar]
- Carter M. Reconsidering overlap-based measures for quantitative synthesis of single-subject data: What they tell us and what they don’t. Behavior Modification. 2013;37:378–390. doi: 10.1177/0145445513476609. [DOI] [PubMed] [Google Scholar]
- Cohen J. Statistical power analysis for the behavioral sciences. (2nd ed.). 1988. [Google Scholar]
- DeProspero A, Cohen S. Inconsistent visual analysis of intrasubject data. Journal of Applied Behavior Analysis. 1979;12:573–579. doi: 10.1901/jaba.1979.12-573. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Fisher WW, Kelley ME, Lomas JE. Visual aids and structured criteria for improving visual inspection and interpretation of single-case designs. Journal of Applied Behavior Analysis. 2003;36:387–406. doi: 10.1901/jaba.2003.36-387. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hahn U, Warren PA. Perceptions of randomness: Why three heads is better than four. Psychological Review. 2009;116:454–461. doi: 10.1037/a0015241. [DOI] [PubMed] [Google Scholar]
- Lakens D. Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests ad ANOVAs. Frontiers in Psychology. 2013;4:Article 863. doi: 10.3389/fpsyg.2013.00863. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Lanovaz MJ, Giannakakos AR, Destras O. Machine learning to analyze single-case data: A proof of concept. Perspectives on Behavior Science. 2020;43:21–38. doi: 10.1007/s40614-020-00244-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Ma H. An alternative method for quantitative synthesis of single-subject researches: Percentage of data points exceeding the median. Behavior Modification. 2006;30:598–617. doi: 10.1177/0145445504272974. [DOI] [PubMed] [Google Scholar]
- Manolov R, Solanas A. Comparing n = 1 effect size indices in presence of autocorrelation. Behavior Modification. 2008;32:860–875. doi: 10.1177/0145445508318866. [DOI] [PubMed] [Google Scholar]
- McKnight SD, McKean JW, Huitema BE. A double bootstrap method to analyze linear models with autoregressive error terms. Psychological Methods. 2000;5:87–101. doi: 10.1037/1082-989X.5.1.87. [DOI] [PubMed] [Google Scholar]
- Nickerson R. The production and perception of randomness. Psychological Review. 2002;109:330–357. doi: 10.1037/0033-295X.109.2.330. [DOI] [PubMed] [Google Scholar]
- Ninci J, Vannest KJ, Wilson V, Zhang N. Interrater agreement between visual analysts of single-case data: A meta-analysis. Behavior Modification. 2015;39:510–541. doi: 10.1177/0145445515581327. [DOI] [PubMed] [Google Scholar]
- Parker RI, Vannest KJ, Davis JL, Sauber SB. Combining non-overlap and trend for single-case research: Tau-U. Behavior Therapy. 2011;42:284–299. doi: 10.1016/j.beth.2010.08.006. [DOI] [PubMed] [Google Scholar]
- Pustejovsky JE. Procedural sensitivities of effect sizes for single-case designs with directly observed behavioral outcome measures. Psychological Methods. 2019;24:217–235. doi: 10.1037/met0000179. [DOI] [PubMed] [Google Scholar]
- Scruggs TE, Mastropieri MA. Summarizing single-subject research: Issues and applications. Behavior Modification. 1998;22:221–242. doi: 10.1177/01454455980223001. [DOI] [PubMed] [Google Scholar]
- Scruggs TE, Mastropieri MA. PND at 25: Past, present, and future trends in summarizing single-subject research. Remedial & Special Education. 2013;34:9–19. doi: 10.1177/0741932512440730. [DOI] [Google Scholar]
- Scruggs TE, Mastropieri MA, Casto G. The quantitative synthesis of single-case research: Methodology and validation. Remedial & Special Education. 1987;8:24–33. doi: 10.1177/074193258700800206. [DOI] [Google Scholar]
- Skinner BF. The behavior of organisms: An experimental analysis. Appleton-Century; 1938. [Google Scholar]
- Voss JL, Federmeier KD, Paller KA. The potato chip really does look like Elvis! Neural hallmarks of conceptual processing associated with finding novel shapes subjectively meaningful. Cerebral Cortex. 2012;22:2354–2364. doi: 10.1093/cercor/bhr315. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Wolery M, Busick M, Reichow B, Barton EE. Comparison of overlap methods for quantitatively synthesizing single-subject data. Journal of Special Education. 2010;44:18–28. doi: 10.1177/0022466908328009. [DOI] [Google Scholar]
- Wolfe K, Seaman MA, Drasgow E. Interrater agreement on the visual analysis of individual tiers and functional relations in multiple baseline designs. Behavior Modification. 2016;40:852–873. doi: 10.1177/0145445516644699. [DOI] [PubMed] [Google Scholar]
- Wolfe K, Seaman MA, Drasgow E, Sherlock P. An evaluation of the agreement between the conservative dual-criterion method and expert visual analysis. Journal of Applied Behavior Analysis. 2018;51:345–351. doi: 10.1002/jaba.453. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
(XLSX 3655 kb)
(XLSX 28 kb)
(XLSX 3655 kb)
(XLSX 603 kb)
(XLSX 1393 kb)
Data Availability Statement
Upon request

