Search and Selection Procedures of Literature Reviews in Behavior Analysis

Abstract

Literature reviews allow professionals to identify effective interventions and assess developments in research and practice. As in other forms of scientific inquiry, the transparency of literature searches enhances the credibility of findings, particularly in regards to intervention research. The current review evaluated the characteristics of search methods employed in literature reviews appearing in publications concerning behavior analysis (n = 28) from 1997 to 2017. Specific aims included determining the frequency of narrative, systematic, and meta-analytic reviews over time; examining the publication of reviews in specific journals; and evaluating author reports of literature search and selection procedures. Narrative reviews (51.30%; n = 630) represented the majority of the total sample (n = 1,228), followed by systematic (31.51%; n = 387) and meta-analytic (17.18%; n = 211) reviews. In contrast to trends in related fields (e.g., special education), narrative reviews continued to represent a large portion of published reviews each year. The evaluated reviews exhibited multiple strengths; nonetheless, issues involving the reporting and execution of searches may limit the validity and replicability of literature reviews. A discussion of implications for research follows an overview of findings.

An ever-increasing volume of material produced through novel research underscores two essential scientific processes: (1) the generation of new information and (2) the incorporation of discoveries into existing knowledge (Chalmers, Hedges, & Cooper, 2002). Across all of the subdisciplines of behavior analysis (e.g., applied behavior analysis [ABA], experimental analysis of behavior [EAB]; Cooper, Heron, & Heward, 2007), researchers appraise previous studies, summarize knowledge, examine scientific trends, and assess the efficiency of interventions through approaches collectively referred to as literature reviews (Slavin, 1995). Collected research findings guide the decisions of researchers, practitioners, and policymakers (Maggin, Talbott, Van Acker, & Kumm, 2017). Researchers assess previous scholarship in accordance with two general approaches: the narrative or systematic review (Petticrew & Roberts, 2008).

The narrative review relies on the subjectivity of authors in identifying relevant material (Cooper, Hedges, & Valentine, 2009). Narrative reviews typically lack formal search or analysis procedures and possess flexibility in terms of questions of interest (e.g., theory; Talbott, Maggin, Van Acker, & Kumm, 2018). Often indistinguishable from discussion articles, narrative reviews in behavior analysis cite examples from the literature to extend or challenge existing theories of behavior (e.g., Shahan, 2010). Narrative reviews also derive appeal as an accessible means of circulating findings (Maggin et al., 2017). Hence, narrative reviews promote the use of research by nontechnical audiences (Behrstock-Sherratt, Drill, & Miller, 2011).

Critics have questioned the utility of narrative reviews as an acceptable means of analyzing research due to their opaque, subjective approach to identifying included studies (Chalmers et al., 2002; Odom, 2009; Slavin, 1995). Hantula (2016) called for reviews to incorporate systematic approaches, arguing narrative reviews have the potential to misrepresent evidence and incorporate bias. The absence of methodological detail further impedes efforts at replication, as when authors expound upon a previous review of a topic (e.g., functional analysis; Schlichenmeyer, Roscoe, Rooker, Wheeler, & Dubé, 2013) or adopt different approaches in examining similar issues (e.g., Lemons et al., 2016).

In contrast, systematic reviews provide transparent descriptions of search and selection procedures (Garg, Hackman, & Tonell, 2008). This approach to evaluating previous literature, though initially created in the first half of the 20th century, emerged in most fields during the late 1970s and provides a clear description of the method used to identify and select studies for the review (Chalmers et al., 2002). Proponents suggest systematic reviews provide a credible means of evaluating interventions, assessing research quality, revealing gaps in knowledge, and guiding future studies (Cook et al., 2015). The utility of systematic reviews stems from transparent descriptions of research objectives and study selection, which permit consumers to evaluate the strength of the author’s conclusions (Maggin et al., 2017).

Meta-analytic reviews likewise involve thorough literature searches, with the additional step of quantifying and statistically combining the study results, examining moderating variables, and arriving at precise conclusions regarding treatment (Petticrew & Roberts, 2008). Meta-analysis represents an approach to examining literature developed only within the last 50 years (Chalmers et al., 2002). Beyond simply averaging study effects, meta-analysis entails statistically combining findings from two or more separate studies to yield (1) individual effect sizes for each study, (2) a summary statistic consisting of the weighted average of effects from each study, (3) a measure reflecting the uncertainty of results (e.g., confidence interval, probability distribution), and (4) a determination of whether studies measure the same quantity (i.e., fixed effects) or effects following a distribution across studies (i.e., random effects; Deeks, Higgin, & Altman, 2019). When combined with adequate search and selection procedures, meta-analysis purports to concisely encapsulate the results of entire bodies of literature.

Due in part to the magnitude of research evidence available in the latter decades of the 20th century, the number of systematic reviews appearing in Scopus and the Web of Science (WOS) databases grew exponentially between 2000 and 2010 (Hansen & Trifkovic, 2013). The evidence-based practice (EBP) movement, which emphasizes the use of methods supported by multiple high-quality studies (American Psychological Association [APA] Presidential Task Force on Evidence-Based Practice, 2006), further explains the increase in systematic reviews (Cook & Cook, 2013). Although not uniformly accepted within behavior analysis (e.g., Pennypacker, 2012; Slocum et al., 2014), the EBP movement has gained traction within related disciplines of special education (Cook & Odom, 2013) and psychology (APA Presidential Task Force on Evidence-Based Practice, 2006). Systematic reviews play a critical role in substantiating and disseminating EBPs throughout the professional community (Moore et al., 2019). As a result, the methodology of systematic reviews has implications for treatment and practice (Mackay, Barkham, Ries, & Stiles, 2003).

Systematic reviews address a wide range of topics aside from EBPs. Review objectives specific to behavior analysis include (1) evaluating the field (e.g., status of women; McSweeney & Swindwell, 1998), (2) assessing the prevalence of specific research practices (e.g., dependent measure assessment; Kostewicz, King, Datchuk, Brennan, & Casey, 2016), (3) historical overviews (e.g., founding of behavior analysis; Morris, Atlus, & Smith, 2013), and (4) questioning established theory (e.g., verbal operants; Gamba, Goyos, & Petursdottir, 2015). Delineated methodology provides insight into the intent and applicability of reviews (King, Davidson, Chitiyo, & Apple, 2020).

Describing procedures in systematic reviews and meta-analyses alleviates confusion resulting from contradictory findings presented in overlapping reviews, or separate reviews pertaining to the same subject (e.g., Siontis, Hernandez-Boussard, & Ioannidis, 2013). Differences in methodology have resulted in discordant reviews on topics including arthritis (e.g., Thorlund, Druyts, Aviena-Zubieta, Wu, & Mills, 2013), social skills interventions (e.g., Wang, Parilla, & Cui, 2013; cf. Ledford, King, Harbin, & Zimmerman, 2018), and the prevalence of replication studies (e.g., Lemons et al., 2016; c.f. Therrien, Mathews, Hirsch, & Solis, 2016). Processes used in identifying and selecting eligible studies partially explain the disparate findings of discordant reviews. Unstated search and selection procedures prevent consumers from identifying biases or sources of conflict with the potential to mitigate the validity of findings.

Of the many benefits of systematic reviews, perhaps the most relevant to behavior analysis concerns their role in providing a transparent record of the replicability and generality of studies unified through any of a number of variables including subjects, interventions, or outcomes of interest (King et al., 2020). Producing similar results following the employment of a method used in an earlier study (i.e., replication) represents a priority for behavior analysts (e.g., Sidman, 1960; Baer, Wolf, & Risley, 1968; Laraway, Snycerski, Pradhan, & Huitema, 2019; Tincani & Travers, 2019). In this article, we review historical material with relevance to systematic and meta-analytic reviews in behavior analysis. We then describe bias associated with literature search and selection procedures. Finally, we review the search and selection procedures reported in systematic and meta-analytic reviews published in journals of behavior analysis.

Systematic Reviews, Meta-Analyses, and Behavior Analysis

The next section provides an overview of scholarship relevant to literature reviews in behavior analysis. First, we briefly describe approaches to experimentation and data analysis within the field (e.g., repudiation of inferential statistics) with the potential to influence the development and assessment of literature reviews. (e.g., Baer et al., 1968; Graf, 1982; Ator, 1999; Perone, 2019). Next, we focus on issues with a more direct relationship to systematic reviews and meta-analyses, such as the popularity of narrative reviews and a debate regarding methodology originally published in Perspectives on Behavior Science (PoBS; Derenne & Baron, 1999; Kollins, Newland, & Critchfield, 1999, Baron & Derenne, 2000; Critchfield, Newland, & Kollins, 2000). We conclude with an overview of more recent writings concerning replication, statistics, and reviews in behavior analysis.

Experimentation and Data Analysis in Behavioral Analysis

Behavior analysis encompasses a series of distinct approaches to scientific inquiry designed to demonstrate relations between the environment and behavior (Sidman, 1960; Baer et al., 1968). Skinner (1938, 1956) firmly established the focus of behavior analysis on controlled, repeated observations of individual organisms and asserted the independence of the science of behavior from psychology’s adherence to group experimentation. Hence, research in behavior analysis often focuses on the internal replication of experimental effects at the individual level through the systematic manipulation of controlling variables (Sidman, 1960).

Behavior analysts have traditionally rejected the statistical analysis used in group experimentation, in which the aggregation of participants in control or treatment conditions (a) masks variability within each group, (b) obscures the change of behavior over time, or (c) distances researchers from variables that control behavior (Baer et al., 1968, Baer, 1977; Ator, 1999; Johnston & Pennypacker 2009). Beginning at least with Skinner’s (1938) publication of cumulative record curves, behavior analysts prefer to describe, both narratively and with descriptive statistics, graphic depictions of response data. Through this process of visual analysis, the detection and determination of clear experimental effects and functional relations occurs without the impediment of formal significance tests (Perone, 1999; Kazdin, 2011). Behavior analysts (e.g., Perone, 1999; Killeen, 2019) have repeatedly criticized null-hypothesis testing, a cornerstone of inferential statistics (e.g., Nickerson, 2000), as an abstruse, arbitrary process that detracts from observations of behavior change. Wariness of statistics also extends to efforts to fuse inferential analysis with traditional behavior analysis—a form of eclecticism Sidman (1960) warned could irrevocably compromise behavior science and produce “a hopeless confusion of basically incompatible data and principles” (p. 54). Many critics objected to techniques beyond the visual analysis of graphic data displays due to the typical issues associated with statistics as well as the propensity of statistics to inflate the effects of interventions (e.g., Baer et al., 1968; Graf, 1982).

Despite the emphasis on internal replication and the rejection of techniques associated with the aggregation of data, behavior analysts recognize the importance of replicating experimental effects externally, beyond individual subjects (Sidman, 1960). In his foundational Tactics of Scientific Research, Sidman (1960) provided many of the guiding principles for single-case design (SCD) experiments—the traditional approach to behavior analytic research often involving a small number of participants—yet explicitly underscored the importance of replication beyond an original subject. According to Sidman, a single experiment replete with multiple demonstrations of effect (e.g., internal replications as obtained through a reversal design) may provide initial evidence of a relationship between variables. However, the significance of such work may only be truly confirmed through subsequent studies. Just as the repeated demonstration of an effect with a single individual contributes to the evidence base regarding the principles of behavior, external reproduction of a study confirms whether anticipated or unanticipated factors might impede replication. Sidman and subsequent authors (e.g., Baer et al., 1968; Baer, Wolf, & Risley, 1987) further noted the importance of exploring the generality of effects, or the extent initially observed relations between variables maintain across settings, organisms, and behaviors. As with replication, Sidman (1960) suggested reproducing effects across a wide range of participants and laboratories resulted in a stronger body of evidence.

Demonstrating external replication and generality, however, requires multiple studies recognized as sharing some sort of similarity despite differences in subjects, settings, or methods (Critchfield et al., 2000). This process implicitly requires some form of aggregation, yet emphasizing commonalities across experiments potentially negates features that might otherwise distinguish one work from another. Though rejected by many behavior analysts, the pursuit of quantitative approaches to SCD stemmed from (1) concerns regarding the reliability of visual analysis (e.g., DeProspero & Cohen, 1979; Hojem & Ottenbacher, 1988; for a review, see Ninci, Vannest, Wilson, & Zhang, 2015) and (2) an interest in synthesizing the effects of intervention studies featuring SCDs (Scruggs, Mastropieri, & Casto, 1987a). Advocates of synthesizing SCDs suggested the process could result in stronger conclusions regarding intervention effects and a greater understanding of factors (e.g., settings, populations) that influence the generality of effects (Gingerich, 1984). The rejection of these techniques, as well as the dueling emphases on (1) internal replication at the individual level and (2) demonstrating the generality and replication of effects across multiple subjects complicates attempts to summarize behavior analytic research. Generalizing about the effects of interventions or variables across studies, without obscuring the characteristics of individual experiments, represents a considerable challenge for reviewers (Critchfield et al., 2000).

Reviews in Behavior Analysis

Leaving aside the objection to statistics within behavior analysis, Sidman’s (1960) writings regarding the importance of external replication and extension, combined with a general emphasis on highly detailed descriptions of research methods (e.g., Baer et al., 1968), would appear to provide some basis for the proliferation of systematic reviews. However, the origins of behavior analysis as a basic science, in which researchers attempted to examine universal principles through highly controlled animal experiments inspired a less systematic alternative to assessing the literature: the narrative review (Mace & Critchfield, 2010). Narrative reviews often attempt to link laboratory studies to natural phenomenon irrespective of their actual relation in reality (Hayes, Blackledge, & Barnes-Holmes, 2001). This approach to analyzing literature involves logically extending findings from the laboratory into compelling, under-researched areas, while also preserving the individual characteristics of thematically unified studies through in-depth, qualitative descriptions (e.g., Salzberg, Strain, & Baer, 1987).

In the face of increasing efforts to quantitatively synthesize SCD research during the 1980s (e.g., Center, Skiba, & Casey, 1985), Salzberg et al. (1987) offered a pointed rejection of quantitative SCD syntheses (e.g., Scruggs et al., 1987a) as well as a robust endorsement of narrative reviews. Salzberg et al. (1987) provided a narrative examination of studies quantified in an earlier synthesis (Scruggs, Mastropieri, Cook, & Escobar, 1986) and concluded the narrative approach preserved information discarded through alternative approaches, commenting “if what we have done here is meta-analysis, we recommend it as a better model; if not, we recommend it anyway” (p. 48). Critics of Salzberg et al.’s (1987) commentary correctly noted the authors did not address—and exemplified—widely recognized issues with the precision, objectivity, and validity of traditional narrative reviews (e.g., Scruggs, Mastropieri, & Casto, 1987b).

A series of articles initially pertaining to a quantitative synthesis in which the authors did not report detailed description of procedures (Kollins, Newland, & Critchfield, 1997) provided an expansive dialogue concerning appropriate methods for summarizing research in behavior analysis. Derenne and Baron (1999) suggested the quantitative comparison of aggregated studies deviated from the emphasis on individual responses needed to observe the controlling influence of variables on behavior. The authors also identified search and selection criteria that result in the inclusion of studies with dissimilar participants, settings, or quality control procedures as a form of aggregation with the potential to distort findings. Rather than calling for the abandonment of systematic reviews and meta-analyses, however, Derenne and Baron acknowledged the limitations of traditional narrative reviews and proposed that the standards for primary behavior analytic research extend to the selection and scrutiny of studies featured in literature reviews.

The remaining entries in the discourse explicitly highlighted the conceptual challenges of reviewing literature in behavior analysis. Kollins et al. (1999) suggested the aggregation of studies through search criteria represented an unavoidable limitation, because all potential methodological choices are in some way compromised. Baron and Derenne (2000) conceded this point, and recommended a transparent approach that would include, categorize, and analyze studies during the review process. In concluding the exchange, Critchfield et al. (2000) argued that the determination of generality of behavioral effects across studies—a fundamental goal of behavior analysis—inescapably involves accumulating studies with dissimilarities. Citing the decisive role research methodology plays in determining outcomes, the authors further suggested that the effects of various approaches to literature review should be examined empirically rather than be assumed. Both groups of authors conceived of a role for systematic and narrative reviews in behavior analysis. Taken together, the series of articles represented a call for greater transparency and methodological heterogeneity in reviews of research in behavior analysis.

Recent Developments and Summary

The impact of previous scholarship on the prevalence of systematic reviews and meta-analyses cannot be determined without an assessment of publication trends in behavior analysis. Examinations of reviews published in the related discipline of special education reveal an increase in systematic reviews and meta-analyses (King et al., 2020), as well as an increase in syntheses of SCD studies employing the percentage of nonoverlapping data (PND) and other quantitative effect sizes (Maggin, O’Keefe, & Johnson, 2011a). The dramatic increase in the use of effect-size metrics and meta-analysis techniques appearing in special education has had a number of implications for SCD research (Pustejovsky & Ferron, 2017). Researchers appear to have accepted the general premise of meta-analysis as applied with group design studies. Newer effect-size metrics (e.g., log response ratio; Pustejovsky, 2015) exceed nonoverlap effects in terms of technical adequacy, and between-case effects permit the aggregation of single-case and group design studies (Shadish, Hedges, Horner, & Odom, 2015). Analysis increasingly allows for the synthesis and evaluation of data within and across studies, facilitating the preservation of data at the individual level in meta-analyses of SCDs (Pustejovsky & Ferron, 2017). Nonetheless, researchers frequently identify problems with the metrics of effect used in meta-analyses as well as an absence of quantitative effects in the primary literature (Laraway et al., 2019).

Recent authors of meta-analyses have attempted to bridge the gap between practices in special education and the historical dictates of behavior analysis. In addition to calculating numerous effect sizes for SCDs, Lanovaz, Turgeon, Cardinal, and Wheatley (2019) analyzed the probability of an experimental effect replicating within ABAB designs in terms of the percentage of apparent effects, as determined through visual analysis. This metric, similar to the success rate metric proposed by Lanovaz and Rapp (2016), does not avoid the issues with aggregation inherent to all systematic reviews identified by Baron and Derenne (2000). Yet the use of visual analysis as the basis of a descriptive summary statistic of an intervention’s effectiveness, as opposed to a quantitative aggregation of time series data, may be more acceptable to behavior analysts. Likewise, Killeen (2019) advocated for the use of Bayesian statistics, where the predicted effectiveness of an intervention stems from the knowledge of its effectiveness in prior applications. Such an analysis avoids the pitfalls of null-hypothesis testing and uses the record of replication documented across studies as the basis for the derivation of effects rather than the aggregation of individual data points.

Various behavior analysis journals observe different standards for literature reviews. The Journal of Applied Behavior Analysis (JABA) introduced brief reviews in article format in 2010 (“Brief reviews”), with submission guidelines requiring authors to evaluate research published in JABA within 5 years of the review and observe strict page limitations. These guidelines potentially limit information available to consumers regarding search and selection procedures. In contrast, PoBS endorsed the use of methodologically explicit reviews featuring sophisticated techniques over narrative reviews—an apparent departure from tradition, potentially serving as a tacit acknowledgement of the persistence of narrative reviews in the field (Hantula, 2016). JEAB likewise encouraged authors to depart from traditional methods of analysis and submit articles with more sophisticated forms of statistics, though not specifically in the context of literature reviews, and always with an eye toward preserving the behavior analytic nature of the journal (Kyonka, Mitchell, & Bizo, 2019; Galizio, 2020).

A general concern with the veracity of research findings has also inspired renewed interest in literature reviews in behavior analysis. Scholars in psychology and other behavioral sciences have increasingly expressed alarm with the apparent failure of many research teams to successfully replicate previously observed findings—a phenomenon often described as a replication crisis (Perone, 2019; Tincani & Travers, 2019). Laraway et al. (2019) suggested the traditional emphasis on replication within behavior analysis does not protect the field from widespread replication failures. Factors contributing to a possible replication crisis in behavior analysis include (1) publication practices that relegate unsuccessful replication studies to obscurity; (2) inconsistent, imprecise descriptions of study procedures and effects; (3) the proliferation of different measurement systems across studies; and (4) the varying quality of experiments implemented by individual research teams. Laraway et al. further identified thorough, transparent literature reviews and meta-analyses as a potential safeguard against the replication crisis. Hence, concerns related to the perceived validity of research have inspired a reevaluation of the unsettled question regarding appropriate methods for summarizing studies in behavior analysis (Baron & Derenne, 2000).

In sum, the importance of preserving individual responses of organisms, combined with the traditional use of narrative reviews, may have prevented the appearance of systematic reviews and meta-analyses within behavior analysis despite the emphasis placed on external replication and generality by prominent figures (e.g., Sidman, 1960). Dismissal of statistical methods employed in meta-analysis showed minimal signs of abeyance by 1990s (e.g., Ator, 1999), though scholars increasingly espoused the benefits explicit systematic literature search and procedures (e.g., Derenne & Baron, 1999). Although the aggregation of studies inherent to all forms of review poses the risk of violating historic precepts of behavior analysis, such risks are essential to the essential task of demonstrating the generality of behavioral principles (Critchfield et al., 2000; Sidman, 1960). Moreover, the careful, transparent application of research methods may mitigate many concerns regarding methods employed in literature reviews.

Potential Sources of Bias in Literature Reviews

As noted by Kollins et al. (1999), decisions authors make in identifying and selecting literature unavoidably influence findings. The next section reviews characteristics of search and selection procedures with the potential to bias conclusions. These include methods used in identifying articles as well as the bases for excluding otherwise eligible studies (e.g., publication status, quality, relevance, methods of analysis).

Search Procedures

Coverage of the literature represents a measure scholars use to assess the quality of reviews (Delaney & Tamás, 2018). Booth’s (2010) rationale for conducting broad, transparent searches included increasing the likelihood of locating pertinent scholarship, demonstrating procedural rigor, and avoiding questions regarding the completeness of the review. Authors who provide details of database searches facilitate efforts to replicate and evaluate limitations of reviews (e.g., Boolean operators; Sampson et al., 2009). Although they represent powerful tools, databases alone cannot support claims regarding the comprehensiveness of a search (Delaney & Tamás, 2018). Using a small number of electronic databases likewise risks omitting references due to the disparity of records featured in specific databases (e.g., Lawrence, 2008). Likewise, hand searches involving discipline-specific journals, when used in isolation, may overlook work published in alternative outlets. Restricted searches potentially omit research outside of the traditions of behavioral analysis that may nonetheless strengthen intervention practices and research methodology in general (Laraway et al., 2019).

Publication Status

Authors select articles for review based on their resources and research objectives (Petticrew & Roberts, 2008). It can be argued that attempting to include all studies, regardless of relevance or quality, risks introducing bias into the review (Booth, 2010). Unlike peer-reviewed articles, authors may encounter difficulty in locating many unpublished sources concerning a topic. Partial inclusion of specific types of research records (e.g., master’s theses) based on the author’s inability to account for a large portion of existing material represents another source of bias (Ferguson & Brannick, 2012). Hence, many reviewers exclude research based on availability, publication source, or quality (Mahood, Van Eerd, & Irvin, 2014).

The tendency of reviews to exclude unpublished work, however, compounds the effect of publication bias—the overrepresentation of studies with positive findings in peer-reviewed journals (Cook & Therrien, 2017; Tincani & Travers, 2019). In other words, journals publish initial findings that demonstrate an effective intervention, and ignore more mundane replication studies, especially when such studies do not result in positive findings. Thus, published studies typically demonstrate positive intervention effects, relative to unpublished studies.

Publication bias partially stems from the predilection of journal editors for ostensibly novel, exciting, or, at minimum, statistically significant results (Laraway et al., 2019). The resistance of behavior analysis to inferential statistics has supposedly inoculated the field from publication bias, because behavior journals typically do not reject articles on the basis of statistical significance (Pustejovsky & Ferron, 2017; Tincani & Travers, 2019). However, the emphasis on large effects in SCD studies (Baer et al., 1968) may have barred studies with ambiguous findings from publication (Kratochwill, Levin, & Horner, 2018). Results of a survey conducted by Shadish, Zelinsky, Vevea, and Kratochwill (2016) indicated SCD researchers preferred studies with positive findings. Further analyses of studies in behavior analysis (e.g., Sham & Smith, 2014) and related fields (e.g., special education; Gage, Cook, & Reichow, 2017) reveal a disparity in the findings of published and unpublished work, suggesting the exclusion of unpublished studies distorts the results of reviews (Polanin, Tanner-Smith, & Hennessy, 2016). When findings differ based on the language of publication, exclusively including English-language sources constitutes an additional source of bias (e.g., Egger et al., 1997).

Study Quality

Study selection based on methodological rigor may also alter the results of systematic reviews. Considerations of study quality primarily relate to efforts taken in reducing the apparent influence of irrelevant factors, or threats to validity, on results (Petursdottir & Carr, 2018). Efforts to introduce research findings into practice (e.g., the EBP movement) often emphasize high-quality studies, because poor studies may produce unwarranted conclusions (Cook & Cook, 2013). Due to concerns regarding the role of clinical judgement, terminology, and other issues related to evidence (e.g., Slocum et al., 2014), institutions within behavior analysis (e.g., Association for Behavior Analysis International [ABAI]) have yet to disseminate formal evidence standards. Nonetheless, in general behavior analysts encourage the evaluation of methodology when consulting research (Petursdottir & Carr, 2018) and have noted factors such as baseline trends and the length of observations as features to consider when evaluating the rigor of studies (e.g., Derenne & Baron, 1999). Many disciplines including psychology (e.g., Weisz & Hawley, 2002) and education (e.g., What Works Clearinghouse [WWC], 2017) have adopted formal indicators of quality often used to eliminate studies from consideration (e.g., Busacca, Anderson, & Moore, 2015). Guidelines for evaluating quality encompass various designs, including experimental, correlational, and qualitative work (e.g., Thompson, Diamond, McWilliam, Snyder, & Snyder, 2005).

The practice of excluding material based on quality poses several issues, however, in particular when the literature base consists of few studies or the area of interest impedes high-quality research (Petticrew, 2015). Failure to satisfy standards does not entirely preclude the contribution of important information to future inquiry. Studies that do not meet quality standards due to the failure to demonstrate a treatment effect may provide insight into when interventions do not yield results (Kratochwill et al., 2018). In addition, the application of quality indicators has implications for reviews beyond their use as a basis for exclusion. Different indicators of quality produce discordant overlapping reviews (Moore, Maggin, Thompson, Gordon, Daniels, and Lang, 2019; Wendt & Miller, 2012). For example, Maggin, Chafouleas, Goddard, and Johnson (2011b) refuted earlier reviews suggesting token economies represented an EBP after determining many studies supporting the practice in educational environments did not meet quality standards.

Relation to Objectives

Decisions regarding selection criteria with regard to research design, intervention, and participant status relate to the objectives of the review (King et al., 2020). Less precise selection criteria permit the inclusion of a broad array of studies, with the consequence of grouping fundamentally dissimilar studies (e.g., “apples to oranges”; Sharpe, 1997). Subsequent conclusions potentially distort views of outcomes, interventions, or other variables (Maggin et al., 2017). Derenne and Baron (1999) identified the conflation of studies with dramatically different participants and interventions as a threat to validity of systematic reviews and meta-analyses in behavior analysis. Reviews relegated to SCDs historically associated with behavior analysis risk failing to account for group designs that provide additional evidence regarding the effectiveness of interventions.

Analysis

Authors of systematic reviews of intervention research often provide a measure of effectiveness beyond those featured in the original study. Because effect sizes preclude a range of common SCDs and group designs (e.g., Shadish et al., 2015), such procedures effectively constitute a criterion for exclusion. Quantitative effect sizes, typically associated with meta-analyses of group-designs, increasingly appear in systematic and meta-analytic reviews of SCD research as a means of supplementing the limitations of visual analysis (Manolov, Losada, Chacon-Moscoso, & Sanduvete-Chaves, 2016). In a departure from the historic reliance on visual analysis and repudiation of statistics in behavior analysis (e.g., Wolery, Busick, Reichow, & Barton, 2010), frequent publishers of SCD research increasingly require reviews to quantify effects (e.g., Hantula, 2016). Because all effect sizes cannot be applied to the range of designs used to examine questions relevant to interventions and other concerns, the types of analyses featured in reviews have implications for findings.

Purpose

Critics suggest systematic reviews present questionable findings beneath a misleading veneer of technical precision (e.g., Slocum et al., 2014; Ioannidis, 2016). Concerns regarding the adverse effects of inaccurate literature searches led to the efforts to improve systematic reviews (Chalmers et al., 2002). Guidelines set forth by the Cochrane Collaboration—an organization dedicated to systematic reviews of healthcare interventions (Cooper et al., 2009)—and the similar Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA; Liberati et al., 2009) identify core aspects of the search process. These guidelines require explicit descriptions of objectives, search procedures, selection criteria, data collection, and study results (Shea, Dube, & Moher, 2001). In addition to disclosing search and selection procedures, reviews satisfying the Cochrane standards feature exhaustive searches and selection criteria designed to minimize bias (Higgins et al., 2011). Efforts to assess literature reviews have also emerged in numerous disciplines (e.g., education; Valentine, Cooper, Patall, Tyson, & Robinson, 2010; Moher, Tetzlaff, Tricco, Sampson, & Altman, 2007). Notwithstanding the considerable disagreement regarding the appropriate approach to summarizing previous studies in behavior analysis, scholars appear to be reaching consensus regarding the importance of methodological transparency in literature reviews (Derenne & Baron, 1999; Critchfield et al., 2000; Laraway et al., 2019).

In a recent study, King et al. (2020) conducted a broad assessment of reviews published in special education from 2004 to 2016 (n = 1,196), emphasizing the search and selection procedures featured in systematic reviews and meta-analyses (n = 901). Findings indicated a marked shift toward transparency, with a decrease in the number of narrative reviews accompanied by an increase in the number of published systematic reviews. Transparent descriptions of searches, which in general featured multiple search methods (73.60%), appeared in the majority of reviews. Authors reported explicit details regarding specific inclusion criteria less frequently. Areas in which reported search criteria represented potential problems included the reliance on peer-reviewed and English-language research as well as the limited use of quality assessment. Given the overlap in the populations served and interventions employed, findings from special education have potential implications for behavior analysts. Nonetheless, the characteristics of reviews published in behavior analysis remain uncertain.

An appraisal of literature reviews in behavior analysis has the potential to promote discourse regarding the strengths and limitations of publication practices. The extent to which literature reviews include a wide range of evidence directly relates to their success in identifying effective treatment, describing research trends, and challenging theory. The characteristics of published systematic literature reviews and meta-analyses further reflect the quality of the information consumed within the field. This study evaluated the characteristics of reviews appearing in behavior analysis journals associated with sources of bias. To reflect recent review practices and assess publication trends, the search identified literature reviews published between 1997 and 2017. Research questions included: (1) How has the appearance of narrative, systematic, and meta-analytic reviews changed over time? and (2) To what extent do authors of systematic reviews and meta analyses provide details regarding search and selection procedures? Given their prominence in the field of behavior analysis, we report findings from all journals constituting the sample as well as the publications of the Society for the Experimental Analysis of Behavior (SEAB) and ABAI (n = 7; Table 1).

Table 1.

Identified Records and Review Methodology of Records from Included Journals

Title*	Impact	Identified Records		Review Methodology			F
		DB	HS	N	S	M
Group 1
Behavioral Disorders	1.161	183	7	8	31	8	47
Behavior Modification	2.024	142	20	40	29	13	82
Journal of Clinical Psychology	2.33	380	14	77	29	23	129
Journal of Consulting and Clinical Psychology	4.536	234	8	24	22	81	127
Psychopharmacology	3.222	210	23	99	45	14	158
Group 1 Total		1149	72	248	156	139	543
Group 2
Behavior Therapy	3.228	120	7	18	12	17	47
Journal of Applied Behavior Analysis	1.534	108	13	35	20	4	59
Journal of Behavioral Education	1.087	67	5	6	15	12	33
Journal of Clinical Psychology in Medical Settings	1.893	153	2	13	20	4	37
Journal of Organizational Behavior Management	1.419	95	6	11	18	1	30
Perspectives on Behavior Scienceb	1.357	98	30	25	29	2	56
Group 2 Total		641	63	108	114	40	262
Group 3
The Analysis of Verbal Behavior	-	39	12	9	19	-	28
Behavior Analysis: Research and Practicea (1999–2017)	-	95	5	19	13	6	38
Behavioural and Cognitive Psychotherapy	1.633	282	7	19	14	5	38
Behavioral Interventions	.792	48	7	14	10	2	26
Journal of Behavior Therapy and Experimental Psychiatry	2.397	74	-	7	2	11	20
Journal of Experimental Psychopathology (2010–2017)	.479	24	15	9	16	1	26
Group 3 Total		562	46	77	74	25	176
Group 4
Behavioural Pharmacology	1.854	143	14	44	3	2	49
Behavior Analysis in Practice (2008–2017)	-	60	3	9	10	-	19
Child and Family Behavior Therapy	.531	206	3	10	4	1	15
Experimental and Clinical Psychopharmacology	2.354	73	5	18	8	3	29
The Psychological Record	1.026	127	7	14	6	-	20
Group 4 Total		609	32	94	31	6	132
Group 5
Behavior and Philosophy	-	36	1	1	-	-	1
Behavior and Social Issues	-	33	2	3	4	-	7
Behavioural Processes	1.555	185	18	65	2	1	68
European Journal of Applied Behavior Analysis (2000–2017)	-	238	4	7	4	-	11
Journal of the Experimental Analysis of Behavior	2.010	110	4	20	2	-	22
Journal of Experimental Psychology-ALC	1.861	16	-	6	-	-	6
Group 5 Total		618	29	102	12	1	115
TOTAL (Across Groups):		3,579	242	630	387	211	1,228

Note: DB = Database search; HS = Hand search; N = Narrative; S = Systematic; M = Meta-analysis; F = Final. Underlined journals published by the Association for Behavior Analysis International or the Society for the Experimental Analysis of Behavior. Database column refers to records retrieved in the initial search. Impact factors taken from the Web of Science (WOS) Journal Citation Reports® Social Sciences Edition (2017). Hand search record included in the final sample. Groups represent quintiles based on the total systematic and meta-analytic reviews. *Journals published within the full search range unless otherwise noted. ^aAlternate titles featured in search include The Behavior Analyst Today. ^bAlternate titles featured in search include The Behavior Analyst

Method

Search Procedures

A literature search completed in February 2018 selected reviews published in peer-reviewed, English-language journals related to behavior analysis. At the suggestion of the editor of PoBS, journals published by the SEAB, ABAI, or identified by Hantula, Critchfield, and Rasmussen (2017) met criteria for inclusion. Journals featured in the 2017 WOS Journal Citation Reports® Social Sciences Edition (JCR; Clarivate Analytics, 2017) categories for clinical psychology or behavioral sciences also met criteria for inclusion, provided their aims and scope explicitly referred to behavior analysis or SCD. WOS subject categories correspond to citation patterns (Wang & Waltman, 2016) and represent an internationally accepted journal subject classification system (Waltman, 2016). Special education journals (e.g., Education and the Treatment of Children) met criteria for exclusion due to their inclusion in a previous review (King et al., 2020). A complete list of included journals (n = 28) appears in Table 1.

For each of the journals, an electronic search evaluated articles published between 1997 and December 2017. We adopted a 20-year search time range for three reasons. First, we accepted the rationale of WWC (2017) reviews, which excludes material published prior to 20 years of the review in order to represent current practice and avoid scholarship involving methods, contexts, or populations substantially different from the contemporary setting. Second, we wanted to avoid charges of penalizing older reviews (i.e., > 20 years old) composed several years prior to the initial refinement of interdisciplinary standards for literature syntheses (e.g., Moher et al., 1999; see Shea et al., 2001). Third, the initial year of eligibility for the current search coincides with the publication of the review conducted by Kollins et al. (1997), which served as the basis of the first series of articles debating the merits of literature syntheses in PoBS (Derenne & Baron, 1999; Kollins et al., 1999, Baron & Derenne, 2000; Critchfield et al., 2000).

Initial search procedures consisted of an electronic search of article titles and abstracts. In particular, a command line search of PsycINFO and ProQuest Psychology databases in the title and abstract search fields used the following Boolean string:

(JN("[JOURNAL]") AND TI(review* OR meta-analy* OR "meta analy*" OR synthes* OR survey)) OR (JN("[JOURNAL]") AND AB(review* OR meta-analy* OR "meta analy*" OR synthes* or survey))

The research team reviewed identified titles and abstracts and eliminated records that were irrelevant to the research questions (e.g., primary investigations). We then evaluated full texts of the remaining abstracts using article inclusion criteria. Following the electronic search, we conducted a hand search of all material published in included journals over the entire period of interest (i.e., 1997–2017).

Inclusion Criteria

Literature reviews, defined as an explicit assessment of previous literature, articles, studies, research, or evidence without describing an original investigation, included articles (1) identified as a literature review, survey, evaluation, meta-analysis, summary, or synthesis in the article title or abstract; and (2) solely analyzed data from aggregated manuscripts, books, reports, or dissertations. Articles satisfying these parameters appearing in selected journals (Table 1) within the targeted publication range met criteria for inclusion regardless of population, intervention, or other features of the targeted studies. Excluded articles reviewed clinical records (e.g., Kahng et al., 2015) or topics without explicitly referencing supporting literature or research. Book reviews, editorials, opinion pieces, and topic reviews not identified as literature reviews also met criteria for exclusion. Five members of the research team, including two PhD-level faculty members and three doctoral students, evaluated all titles and abstracts.

Coding

The research team assessed articles using binary codes derived from previous evaluations (e.g., King et al., 2020; Maggin et al., 2017). Prior to application, an expert panel consisting of three PhD faculty members with experience conducting reviews and behavior analysis evaluated the codes. Coders included three PhD-level faculty and four doctoral students in behavior analysis or special education. All coders received training via a 60-min video-recorded module and practiced coding on a minimum of three sample articles until reaching criterion of 95.00% agreement on three consecutive articles. Table 2 provides a detailed description of codes.

Table 2.

Coding Categories and Descriptions

Indicator	Sub-Categories	Description
Methodology	Narrative	Review does not provide information regarding search procedures.
	Systematic	Review completely or partially identifies procedures used to identify studies.
	Meta-analysis	Systematic reviews described as meta-analyses or that statistically analyzed results.
Area of Focus	Intervention	Review concerned treatment or intervention in terms of their outcomes.
	Feature	Review primarily assessed features or characteristics of studies (e.g. fidelity).
	Other	Review addressed conceptual, historical, or other concerns.
Search Procedures	Database	Indicated use of a database search and number of databases used. Also included reports of search terms including: provided (specific terms); field codes (i.e., terms searched in a specific component of the article); and Boolean operators (i.e., explicit syntax used in the search).
	Hand Search	Indicated a manual search of publications. Additional codes noted whether authors specified the publications searched and whether search of encompassed a single journal or the journal that published the review exclusively.
	Ancestral	Authors reported searching reference lists or literature reviews for articles.
	Consultation	Authors consulted researchers in the field or editors to identify eligible articles.
Selection Criteria	Temporal	Article identified date of occurrence (i.e., when the search was conducted) and date range (i.e., years in which articles were eligible for inclusion).
	Language	Indication of explicit language restriction (i.e., whether language served as basis of exclusion), nonexplicit restriction (i.e., exclusive search of English-language publications) and English-only (i.e., explicit restriction of inclusion to English publications).
	Operational Criteria	Indicated if review concerned specific age groups, disabilities, behaviors, or interventions. Not applicable for reviews unrelated to these factors. If applicable, noted whether authors provided operational definitions for each.
	Sources Reviewed	Provides indication of documents explicitly subjected to analysis. Included peer-reviewed journals, dissertations, book chapters, and other (e.g., conference proceedings). Also indicates whether authors exclusively included peer-reviewed sources or articles exclusive to a single journal.
	Designs	Identified designs of studies included in review. Includes group, qualitative, single-case, not specified, and other (e.g., conference proposals).
	Quality	Authors evaluated and/or excluded studies based on formal assessment (i.e., explicitly outlined or cited quality indicators). Categories for specific indicators (e.g., author created, What Works Clearinghouse) developed following analysis of reported citations.
	Analysis	Refers to procedures used to evaluate effects of interventions; visual analysis (i.e., explicit procedures used to interpret or categorize data), author report (i.e., presents claims provided by study authors or does not specify means of analysis), and effect size. Categories for specific effect indices developed following analysis.

Descriptive codes

Coders recorded the date of publication and first author of all articles.

Review classification

Classification codes categorized reviews as narrative, systematic, or meta-analytic. Following principal data collection, the combined total of systematic and meta-analytic reviews in each publication served as the basis for dividing journals into quintiles.

Area of focus

Area of focus codes reflected the topics addressed by the authors. Areas of focus included interventions, features, and other. Given the critical role of systematic reviews in identifying evidence supporting the use of specific treatments, we reported codes pertaining to intervention reviews separately.

Search procedures

Search procedure codes denoted the use of (1) electronic databases, (2) hand searches, (3) ancestral searches, and (4) expert consultation. Database searches included any procedure involving the use of electronic programs designed to search article features such as comprehensive indices of published material (e.g., PsycINFO), online applications (e.g., Google Scholar), and search engines directly aligned with a journal (e.g., PoBS). Due to the availability of journal-specific databases, reviews that selected articles from a single journal without identifying the method did not receive a search procedure code. Additional codes pertained to details of specific search methods (e.g., search terms, number of journals searched).

Selection procedures

Selection procedure codes pertained to explicit reports of article inclusion criteria, including publication date, language (e.g., English only), operational descriptions of targeted variables, eligible sources, study designs, quality, and data analysis. Explicit statements represented a definitive indication of the research designs subject to review and did not include general terms (e.g., “experimental”) encompassing various designs. Application of the operationalization code varied based on the foci of the review, which coders assessed across specific dimensions (e.g., age) prior to assessing the suitability of descriptions.

Quality assessment procedures expressly evaluated the quality, rigor, or suitability of design features of included studies. Design features included the presentation of baseline data, data collection frequency, or phases required for experimental control. Quality assessment did not include studies evaluating a single methodological feature across studies (e.g., interobserver agreement [IOA]; e.g., Kostewicz et al., 2016). Coders categorized quality assessment procedures based on reported citations following the completion of the initial analysis. We attributed procedures described in detail without an accompanying citation to the author of the review. Analysis codes indicated whether reviews, regardless of focus, reported effectiveness of included studies (i.e., change in an intervention outcome). As with quality procedures, coders categorized effect indices based on the descriptions featured in the reviews (e.g., PND). Coders categorized similar effect sizes based on the standardized mean difference (SMD) such as Cohen’s d and Glass’s method (Cummings, 2011) as SMD.

Analysis

Given the descriptive nature of study objectives, analyses involved reporting the number or proportion of reviews exhibiting specific characteristics. Authors entered all codes into a spreadsheet during the coding process. Descriptive statistics were calculated using SPSS.

Interobserver Agreement

Coders collected IOA for review selection and coding. Personnel received training prior to both procedures. Coders then evaluated articles published beyond the range of the search (e.g., prior to 2006) until obtaining 95.00% agreement on three consecutive articles. For abstract searches, coders derived agreement—concurrence on inclusion status—by dividing the number of agreements by the total number of abstracts. Two coders independently assessed 100% of abstracts (n = 3,579) obtained via databases, with an IOA of 94.87%. Following the hand search, coders then derived IOA for 20.03% of included articles (n = 246) using the point-to-point approach (Kennedy, 2005), resulting in an average IOA of 95.06% (range = 78.00–100%; SD = 6.4). Authors discussed discrepancies until reaching consensus.

Results

The database search generated 3,579 abstracts following the removal of duplicates. A hand search of all included journals yielded an additional 242 articles. Articles identified in the hand search typically described the study using terms omitted from the database due to their potential to increase the number of irrelevant records (e.g., “evaluate”). In addition, several studies evaluated specific article characteristics without identifying the methodology (i.e., literature review) or compared secondary analysis techniques (e.g., effect sizes for SCDs) without explicitly referring to a requisite literature review in the abstract. The two searches resulted in a total sample of 1,228 reviews. Figure 1 depicts the study selection process.

Fig. 1.

Study selection procedures

On average, 44 reviews appeared in each journal (range = 1–158; SD = 38.53), with an average of 60 reviews published per year (range = 29–114; SD = 19). Each journal published an average of 2.23 reviews per year, (range = 0–16; SD = 2.61). Unless otherwise indicated, the term systematic review in the following sections refers to both systematic and meta-analytic reviews.

Review Classification

Narrative reviews represented the majority of reviews (51.30%; n = 630), with systematic (31.51%; n = 387) and meta-analytic reviews (17.18%; n = 211) appearing less frequently. Narrative, systematic, and meta-analytic reviews exhibited a yearly publication rate of 30 (range = 19–41; SD = 5.88), 18.43 (range = 4–51; SD = 11.24), and 10.05 (range = 1–22; SD = 5.35), respectively. A description of the reviews featured in each journal appears in Table 1. Journals published an average of 22.50 narrative (range = 1–99; SD = 23.39), 14.88 systematic (range = 0–45; SD = 11.09), and 10.55 meta-analytic reviews (range = 0–81; SD = 17.71). A large portion of systematic reviews (88.89%) appeared in the top 3 quintiles (M = 26.62%; range = 19.12–40.31%; SD = 10.60), which encompass 17 of included journals. In contrast, 65.56% of meta-analyses appeared in journals in the top quintile.

Figure 2 depicts the types of reviews published from 1997 to 2017. Though variable, narrative reviews exhibited a declining trend between 1997 and 2004, yet consistently represented over half of reviews published each year until 2006. The percentage of narrative reviews remained largely consistent between 2005 and 2010 before assuming a decelerating trend in 2011. Compared to narrative reviews, fewer systematic reviews appeared from 1997 to 2015. However, systematic reviews exhibited an increasing trend, which rapidly accelerated beginning in 2011. The percentage of systematic reviews published each year surpassed narrative reviews in 2016. Beginning in 1997, meta-analytic reviews exhibited a gradually increasing trend. Although the percentage of meta-analytic reviews in general did not exceed alternatives during this span, meta-analyses and systematic reviews combined represented more than half of all reviews published each year after 2012.

Fig. 2.

Narrative, systematic, and meta-analytic reviews identified in the total sample of behavior journal (top) and the publications of SEAB and ABAI (bottom). Graphs depict the percentage of reviews with all journal, by type, as well as the real number of reviews published each year (gray bar)

SEAB and ABAI publications

Narrative reviews represented 54.50% (n = 115) of the 211 reviews published in SEAB and ABAI journals. Systematic reviews represented slightly less than half of reviews (42.65%; n = 90). Meta-analyses appeared in a small number of reviews (2.84%; n = 6). Yearly publication rates of narrative reviews (5.47; range = 0–10; SD = 3.01) exceeded those of systematic (4.28; range = 1–12; SD = 2.67), and meta-analytic reviews (.29; range = 0–2; SD = .56).

When highlighting the instances of reviews specifically within SEAB/ABAI journals, both similarities and differences appear (Figure 2). Narrative reviews equaled or exceeded the percentage of systematic reviews published in all but 5 years of the assessed publication range. Narrative and systematic reviews maintained consistent levels with a slightly increasing trend and low variability. Prior to 2014, meta-analyses did not appear often within the identified sample. Unlike the overall sample, narrative and systematic maintained approximately the same level and meta-analyses rarely appeared.

Area of Focus

Roughly half of systematic reviews involved intervention research (e.g., Seubert, Fryling, Wallace, Jiminez, & Meier, 2014; 48.83%; n = 292). Conceptual (e.g., Littell & Girvin, 2002; 30.10 %; n = 180) and methodology reviews (e.g., Buchanan, Husfeldt, Berg, & Houlihan, 2008; 21.07%; n = 126) reviews appeared less often. Table 3 describes the search and selection procedures reported in intervention reviews.

Table 3.

Search and Selection Procedures Featured in Systematic Reviews and Meta-analyses

Feature	All Foci (%)		Intervention Focused (%)
	Total	ABAI/SEAB	Total Reviews	ABAI/SEAB
Systematic	387 (64.71)	90 (93.75)	143 (48.97)	19 (76.00)
Meta-Analysis	211 (35.28)	6 (6.25)	149 (51.02)	6 (24.00)
Search Procedures
Methods
One Method	166 (27.76)	55 (57.29)	43 (14.73)	10 (40.00)
Two Method	289 (48.33)	34 (35.42)	157 (53.77)	12 (48.00)
> Three Methods	143 (23.91)	7 (7.29)	92 (31.51)	3 (12.00)
Database Searches
Reported	531 (88.80)	62 (64.58)	283 (96.92)	22 (88.00)
Database Only	107 (20.15)	22 (35.48)	38 (13.42)	7 (31.82)
Unlisted	13 (2.45)	1 (1.61)	7 (2.47)	0 (0.00)
One Database	114 (21.47)	21 (33.87)	40 (14.13)	4 (18.18)
Multiple	404 (76.08)	40 (64.51)	236 (83.39)	18 (81.82)
Mean Databases (R;SD)	3 (1–16; 2)	2.41 (1–7; 1.52 )	3.3 (1–16; 2.24)	2.77 (1–6; 1.51)
Terms Reported	460 (86.63)	47 (75.81)	248 (87.63)	20 (90.91)
Field Reported	275 (51.79)	31 (50.00)	151 (53.36)	12 (54.55)
Booleans Reported	71 (13.37)	2 (3.23)	44 (15.55)	2 (9.09)
Hand Searches
Reported	164 (27.42)	46 (47.92)	62 (21.23)	4 (16.00)
Hand Search Only	45 (27.44)	30 (65.22)	3 (4.84)	3 (75.00)
Publications Listed	143 (87.20)	41 (89.13)	56 (90.32)	4 (100)
Mean Journals Searched (R; SD)	4.9 (1–25; 4)	3.44 (1–17; 4.22)	4.9 (1–25; 4.24)	2.5 (1–7; 3)
Restricted to Single Journal	30 (18.29)	18 (39.13)	3 (4.84)	2 (50.00)
Exclusive to Same Journal	24 (14.63)	15 (32.61)	2 (3.23)	2 (50.00)
Ancestral Searches
Reported	395 (66.05)	31 (32.29)	232 (79.45)	13 (52.00)
Ancestral Search Only	12 (3.29)	3 (9.68)	2 (.86)	0 (0.00)
Consultation
Reported	97 (16.22)	5 (5.21)	64 (21.92)	4 (16.00)
Consultation Only	1 (1.03)	0 (0.00)	0 (0.00)	0 (0.00)
Selection Procedures
Temporal
Date of Search	95 (15.89)	6 (6.25)	50 (17.12)	2 (8.00)
Range of Search	283 (47.32)	60 (62.50)	121 (41.44)	13 (52.00)
Language
Explicit Restrictions	216 (36.12)	9 (9.38)	115 (39.38)	4 (16.00)
Non-explicit Restriction	68 (11.37)	36 (37.50)	6 (2.05)	4 (16.00)
English Only	250 (88.03)	43 (95.56)	102 (84.30)	8 (100)
Operational Criteria
Applicable	450 (75.25)	43 (44.79)	284 (97.26)	24 (96.00)
All Concerns Operationalized	237 (52.67)	24 (55.81)	143 (50.35)	13 (54.17)
Sources Reviewed
Explicit	204 (34.11)	33 (34.38)	112 (38.36)	13 (52.00)
Peer-reviewed	190 (93.14)	27 (81.82)	111 (99.11)	13 (100)
Peer-reviewed Exclusive	155 (75.98)	21 (63.64)	89 (79.46)	11 (84.62)
	Total	ABAI/SEAB	Total	ABAI/SEAB
Dissertation	28 (13.73)	4 (12.12)	20 (17.86)	2 (15.38)
Book Chapter	11 (5.39)	6 (18.18)	7 (6.25)	0 (0.00)
Other	18 (8.82)	3 (9.09)	18 (16.07)	0 (0.00)
Design
Explicit	222 (37.12)	21 (21.88)	155 (53.08)	13 (52.00)
Group	156 (70.27)	9 (42.86)	115 (74.19)	5 (38.46)
Group Exclusive	115 (51.80)	3 (14.29)	87 (56.13)	3 (23.08)
Single-case	89 (40.09)	17 (80.95)	63 (40.64)	10 (76.92)
Single-case Exclusive	52 (23.42)	11 (52.38)	37 (23.87)	8 (61.54)
Qualitative	14 (6.31)	3 (14.29)	7 (4.52)	0 (0.00)
Other	28 (12.61)	4 (19.05)	10 (6.45)	1 (7.69)
Quality
Formal	114 (19.06)	1 (1.04)	90 (30.82)	1 (4.00)
Cochrane	18 (15.79)	0 (0.00)	16 (17.78)	0 (0.00)
APA Task Force	4 (3.51)	0 (0.00)	3 (3.33)	0 (0.00)
CEC/WWC*	24 (21.05)	1 (100))	20 (22.22)	1 (100)
Author	22 (19.30)	0 (0.00)	16 (17.78)	0 (0.00)
Other	51 (44.74)	0 (0.00)	42 (46.67)	0 (0.00)
Multiple	17 (14.91)	0 (0.00)	14 (15.56)	0 (0.00)
Excluded	8 (7.02)	0 (0.00)	7 (7.78)	0 (0.00)
Analysis
Assessed Outcomes	481 (80.43)	42 (43.75)	292 (100)	25 (100)
Author Report	290 (60.29)	30 (71.43)	156 (53.42)	14 (56.00)
Author Report Only	223 (46.36)	26 (61.90)	108 (36.99)	12 (48.00)
Visual Analysis	27 (5.61)	4 (9.52)	20 (6.85)	3 (12.00)
Effect Sizes Calculated	238 (49.48)	11 (26.19)	168 (57.53)	9 (36.00)
Multiple Effect Sizes	46 (19.33)	3 (27.27)	33 (19.64)	3 (33.33)
SMD	152 (63.87)	8 (72.73)	119 (70.83)	6 (67.00)
Pearson’s r	27 (11.34)	0 (0.00)	5 (2.98)	(0.00)
PND	19 (7.98)	2 (18.18)	15 (8.93)	2 (22.22)
Tau-U	12 (5.04)	1 (9.09)	6 (3.57)	1 (11.11)
NAP	7 (2.94)	1 (9.09)	3 (1.79)	1 (11.11)
PAND	5 (2.10)	0 (0.00)	4 (2.38)	0 (0.00)
IRD	5 (2.10)	0 (0.00)	4 (2.38)	0 (0.00)
Other	61 (25.63)	1 (9.09)	41 (24.40)	1 (11.11)

Note. Percentages exceeding 100% have articles with several characteristics. Percentages of indented, italicized, or underlined items derived from previous category without indentation. ABAI/SEAB = Journals published either by the Association for Behavior Analysis International or the Society for the Experimental Analysis of Behavior; SMD = standardized mean difference

SEAB and ABAI publications

Systematic reviews appearing in SEAB and ABAI publications in general involved reviews of methodology (45.83%; n = 44). Conceptual reviews (28.13%; n = 27) and intervention reviews (26.04%; n = 25) represented a roughly equivalent proportion of the remaining articles.

Search Procedures

Systematic and meta-analytic reviews provided descriptions of search procedures. Table 3 describes the number of methods employed, the use of specific search methods, and procedures unique to each method. Authors employed more than one search method in 72.24% of reviews (n = 432), with intervention reviews using multiple searches in 85.27% of instances (n = 249). Authors frequently provided specific databases and search terms. However, information regarding field codes or Boolean operators appeared less frequently. Reviews featuring hand searches in general provided information regarding the publications included in the search. Of the articles featuring hand searches, 27.44% (n = 45) employed the method in isolation. Authors also reported restricting hand searches to a single journal (18.29%; n = 30), though this occurred less frequently in intervention reviews (4.84%; n = 3).

SEAB and ABAI publications

Authors of reviews appearing in SEAB or ABAI journals reported using one search method in the majority of reviews (57.29%; n = 55). Intervention reviews typically featured multiple search methods (60.00%; n = 15). Of these, hand searches represented the most common method used in isolation (54.55%; n = 30), followed by database searches (40.00%; n = 22). Information regarding database searches typically included search terms and fields, though specific Boolean operators appeared less commonly. Approximately one-third of systematic reviews employing hand searches restricted the search to a single journal (39.13%; n =18) or the journal in which the article appeared (32.61%; n = 15).

Selection Procedures

Selection criteria codes pertained to seven domains relevant to the selection of articles within systematic reviews. Domains included criteria related to temporal characteristics (e.g., publication date), language, descriptions of targeted variables, eligible sources, and study designs featured in each review. Codes also pertained to procedures with the potential to influence article inclusion; in particular, the use of quality assessment and methods of data analysis. Table 3 provides an overview of these domains within systematic reviews.

Temporal

Authors described the range of years eligible for review in 47.32% (n = 283) of all reviews and 41.44% (n = 121) of intervention reviews. Search dates appeared infrequently in the total and intervention reviews, at 15.89% (n = 95) and 17.12% (n = 50), respectively.

• SEAB and ABAI publications. The range of years eligible for review appeared in 62.50% (n = 60) of all reviews and 52.00% (n = 13) on intervention reviews. Less than 10.00% of reviews, regardless of focus, reported the date of the search.

Language

Approximately one-third of reviews (36.12%; n = 216) reported exclusion criterion on the basis of language, with 11.37% (n = 68) providing sufficient detail to indicate limits on the basis of language. These reviews typically restricted inclusion to articles written in English (Table 3).

• SEAB and ABAI publications. Reviews appearing in SEAB or ABAI publications infrequently reported explicit criterion concerning the language of eligible publications (9.38%; n = 9). Sufficient information to discern the language of included publications appeared in 37.50% of systematic reviews. Of these, in general authors limited inclusion to English-language publications (95.56%; n = 43).

Operationalization of criteria

The majority of total systematic reviews (75.25%; n = 450) targeted a combination of variables related to participant age, disabilities, behaviors, or interventions. Of these, approximately half operationalized the relevant variables (Table 3).

• SEAB and ABAI publications. Of reviews that targeted applicable variables (44.79%; n = 43), 55.81% operationalized all variables relevant to the search (n = 24).

Sources reviewed

Authors reported explicit descriptions of article sources in 34.11% (n = 204) of total reviews. Descriptions of source criteria appeared more frequently in systematic reviews of intervention research (38.36%; n = 112). Regardless of whether they concerned interventions, systematic reviews most frequently obtained records from peer-reviewed articles. Of these, 75.98% (n = 155) exclusively evaluated peer-reviewed articles. Intervention reviews also excluded sources other than peer-reviewed articles (79.46%; n = 89).

• SEAB and ABAI publications. Articles featured descriptions of eligible sources in 34.38% of the total sample (n = 33) and 52.00% of intervention reviews (n = 12). Searches relied on peer-reviewed articles extensively, with majorities of the total sample (63.64%; n = 21) and intervention reviews (84.52%; n = 11) exclusively featuring peer-reviewed articles.

Study designs

Explicit criteria pertaining to study design appeared in 37.12% (n = 222) of total reviews and 53.08% of intervention reviews (n = 155). Authors reported including group designs and SCDs most frequently (Table 3). Of the reviews that reported design criteria, approximately half of all systematic reviews (51.80%; n = 115) and intervention-focused reviews (56.13%; n = 87) exclusively included group designs. Authors restricted article inclusion to SCDs in 23.42% (n = 52) of total reviews and 23.87% (n = 37) of intervention reviews.

• SEAB and ABAI publications. Within SEAB and ABAI publications, inclusion criteria pertaining to study design appeared in 21.88% (n = 21) of the total sample and 52.00% (n = 13) of intervention reviews. Of these, authors restricted article inclusion to SCDs in 52.38% (n = 11) of the total sample and 61.54% (n = 8) of intervention reviews.

Quality assessment

Formal quality assessment procedures appeared in 19.06% (n = 114) of systematic reviews and 30.82% (n = 30.8) of intervention-focused reviews. Quality assessment included indicators associated with education (e.g., Horner et al., 2005; WWC, 2017; Council for Exceptional Children, 2014), the Cochrane Collaboration (e.g., Higgins et al., 2011), and author created measures. Few reviews (7.02%; n = 8) excluded articles based on formal quality assessment.

• SEAB and ABAI publications. A single review appearing in SEAB and ABAI publications (1.04%) formally assessed the quality of included articles using a contemporary version of the WWC (2017) standards (Sham & Smith, 2014).

Analysis

Authors assessed study outcomes in 80.43% (n = 481) of total reviews and 100% (n = 292) of reviews focusing on interventions. Author report appeared in more than half of the total sample (60.29%; n = 290) and intervention reviews (53.42%; n = 156). Approximately half of all reviews (46.36%; n = 223) used author report exclusively. Visual analyses appeared in 5.61% of reviews (n = 27). Effect sizes appeared more commonly (49.48%; n = 238). Effects indices based on the SMD represented the most frequent effect size (63.87%; n = 152). Nonoverlap effect sizes associated with SCD (e.g., PND, Tau-U; Parker, Vannest, & Davis, 2011) appeared in 20.17% of total reviews (n = 48).

• SEAB and ABAI publications. Outcome assessment appeared in 43.75% (n = 42) of reviews published in SEAB and ABAI journals. Author report represented the only form of analysis in 61.90% (n = 26) of all systematic reviews and 48.00% (n = 12) of intervention reviews. Visual analysis appeared in 9.52% (n = 4) of all systematic reviews. Authors provided effect sizes in 26.19% (n = 11) of reviews.

Discussion

This article evaluated the characteristics of literature reviews published in journals of behavior analysis from 1997 to 2017. Narrative reviews continue to represent a significant portion of published reviews, particularly in SEAB and ABAI publications in which meta-analyses rarely appear. Within the larger sample of journals, the appearance of systematic and meta-analytic reviews have gradually accelerated since the late 1990s. Over the last 5 years, systematic approaches to literature reviews represented more than half of published reviews. Authors of systematic reviews generally conducted comprehensive searches featuring multiple methods, though SEAB and ABAI journals reported using one search method—hand searches—more frequently relative to the total sample. Descriptions of database searches suggested many authors used a range of databases but fewer authors provided details needed to replicate the review (e.g., field codes, Boolean operators). In terms of article selection, intervention reviews provided more explicit descriptions of criteria. Nonetheless, a limited number of reviews and meta-analyses described selection procedures in detail. The described selection procedures in some reviews had the potential to adversely influence results (e.g., reliance on peer-reviewed material) by limiting the number of applicable articles subject to review, and thus reducing the accuracy of findings.

Notwithstanding the emphasis on the behaviors of individuals in behavior analysis, the necessity of examining replication and generality across studies as articulated by Sidman (1960) implicitly entails the evaluation of studies united by some theme. Literature reviews provide a formal mechanism for the documenting the extent of generality across studies. The increasing appearance of reviews over time, regardless of methodology, in general reflects a corresponding commitment to examining previous literature within ABAI and SEAB journals as well as behavior analysis more generally. Debates regarding appropriate approaches toward examining the behavior analytic literature (e.g., Salzberg et al., 1987; Kollins et al., 1999) further suggest the presence or absence of specific features in reviews result from decisions regarding the literature summary process rather than a lack of concern for reviews in general.

Compared to the related discipline of special education (King et al., 2020), reviews appeared ~50% less often in behavior analysis journals. The disparity likely represents the continued emergence of behavior analysis as an independent discipline rather than a lack of adequate reflection on findings within the field (Vargas, 1987; Dorsey, Weinberg, Zane, & Guidi, 2009). Given considerable overlap exists between ABA and services provided to individuals with disabilities (Marr, 2017) as well as psychology (Fraley & Vargas, 1986), much work pertaining to behavior analysis may have appeared in outlets more directly related to these fields. However, the increased appearance of work relevant to behavior analysts in theoretically eclectic disciplines (e.g., special education) may prevent such scholarship from reaching consumers of journals exclusive to behavior analysis (Odom, 2009).

In contrast to the dramatic decline observed in special education (King et al., 2020), narrative reviews continue to represent a large portion of reviews published in behavior analysis and within SEAB and ABAI journals in particular. This represents an apparent persistence of the traditional preference for narrative reviews in behavior analysis (e.g., Salzberg et al., 1987; Mace & Critchfield, 2010) as well as developments unique to related disciplines. Special education falls under the purview of governmental entities that encourage systematic reviews of intervention research in pursuit of EBP (Cook & Cook, 2013). Hence, publication trends may reflect the difference between a primarily applied discipline (i.e., special education) versus a field characterized by a range of nonclinical subdisciplines (e.g., EAB; philosophical behaviorism; Cooper et al., 2007). Flagship applied and cross-categorical journals (e.g., JABA; PoBS) published larger numbers of systematic reviews relative to outlets concerning EAB (e.g., The Psychological Record), practitioner concerns, or other subdisciplines. To reach a wider audience, authors may publish reviews of basic scientific research in cross-categorical journals. Given that fewer than half of all systematic reviews addressed intervention research, however, this does not fully explain the disparity in the publication rates of journals.

The influence of historical predispositions within behavior analysis remains especially apparent in terms of the relative absence of meta-analyses, in particular in ABAI and SEAB journals. Many authors expressed misgivings about the use of meta-analyses in behavior analysis (e.g., Salzberg et al., 1987; Derenne & Baron, 1999), and significance tests and statistical aggregation do not accord with methods espoused by the field’s dominant figures (e.g., Skinner, 1938; Sidman, 1960). In contrast to the total sample, in which the largest numbers of reviews pertained to interventions, many of the systematic reviews featured in ABAI and SEAB journals concerned methodological issues. Leaving aside the value of such work to the field, this suggests authors may have attempted to circumvent disputes pertaining to the summation of research by abstaining from analysis entirely (Critchfield et al., 2000). A related, and potentially more alarming, issue concerns the use of author report to assess intervention results as opposed to any form of objective evaluation—including visual analysis. Avoiding the challenge of summarizing literature in behavior analysis by defaulting to imprecise, narrative descriptions potentially obscures the magnitude and direction of findings and stymies efforts to assess replicability and generality (Scruggs et al., 1987b; Laraway et al., 2019).

Although initially published less frequently than narrative reviews, systematic reviews and meta-analyses increased over time, and combined represent the majority of reviews appearing in the total sample since 2012. The concentration of meta-analyses in a small number of journals, together with the few meta-analyses in SEAB and ABAI publications, suggests these increases may not reflect trends in the field. Regardless, the growth in systematic reviews would appear to reflect a greater emphasis on transparency in behavior analysis in general (Tincani & Travers, 2019). However, systematic reviews and meta-analyses exhibited marked variability in terms of methodological transparency. Authors frequently provided information related to the breadth of the search (e.g., method used), yet uncertainty concerning the years included in the search or precise search terms could lead to difficulties in replicating a review. As noted by Critchfield et al. (2000), methods employed by authors play a large role in determining the results. Whatever the view of individual scholars on the propriety of specific search, selection, or analysis procedures, consumers cannot adequately interpret findings without a clear description of procedures.

In addition to issues with transparency, many reviews in behavior analysis reported selection criteria with the potential to distort findings, such as the exclusion of material other than peer-reviewed journal articles (Cook, 2014). As noted by Sham and Smith (2014), neglecting unpublished literature has negative implications for consumers of behavior analysis due to the inflation of treatment effectiveness associated with publication bias (Cook, 2014). Laraway et al. (2019) likewise identified the emphasis on published material as a threat to the accurate evaluation of replication in behavior analysis. Increasing the accessibility of unpublished material through the inclusion of such studies in widely disseminated literature reviews represents a potential step in reducing the harm associated with ineffective treatment (Sham & Smith, 2014). However, publication practices in behavior analysis currently resemble those in other fields (e.g., special education; King et al., 2020; Gage et al., 2017) that overwhelmingly predicate conclusions regarding treatment effectiveness on peer-reviewed research.

The appearance of quality indicators in behavior analysis journals, though infrequent, approximated their application in fields more closely associated with EBP (i.e., special education; 27.40%; King et al., 2020). Authors most often used quality indicators appearing in earlier, largely uncited work (i.e., “other”), created their own standards, or adopted standards from medicine (i.e., Cochrane Collaboration; Higgins et al., 2011) and education (e.g., WWC, 2017). The use of objective, delineated quality standards represents a positive development as practitioners routinely invoke evidence as (1) a purported benefit of behavior analysis and (2) a justification for punitive interventions (Vollmer et al., 2011). The validity of such claims remains questionable in the absence of a clear definition of acceptable evidence. Consensus regarding evidence standards within behavior analysis could also prevent the confusion created by the preponderance of quality standards (Wendt & Miller, 2012).

Although an exhaustive discussion of data analysis rests beyond the purview of this article, present findings raise issues unrelated to search or selection procedures. Prevalent throughout the reviews, author report of study results represents one the most inclusive forms of assessing outcomes as reviewers include all records regardless of their compatibility with the analysis. Extensive use of this practice appears to reflect an uncritical acceptance of an author’s interpretation, however, which the actual data may not consistently support. Notwithstanding the inclusion of journals publishing a wide variety of studies, the use of SMD indicates that behavior analysis encompasses a range of methodologies beyond SCD. The tradition of SCD research within behavior analysis remains evident given the relative frequency of nonoverlap effect sizes used in the literature (e.g., Parker et al., 2011). Yet the absence of visual analysis invites questions regarding the transparency of this technique. In describing studies, authors may have implicitly applied visual analysis, thereby rendering these conclusions indistinguishable from those of the primary study. Increasing recognition regarding the limitations of visual analysis suggests more transparent applications of the procedure should appear in reviews and primary studies (Manolov & Vannest, 2019).

The logistics of both the publication and research processes may partially explain the acceptance of reviews with ostensible sources of bias. Attempts to provide comprehensive descriptions of search procedures, though needed to evaluate the review, often conflict with journal page limitations. Relying exclusively on English-language research potentially poses threats to the validity of findings and underscores issues behavior analysis continues to encounter in becoming an international discipline (Martin, Nosik, & Carr, 2016). The provincial nature of certain questions, combined with the difficulty in securing and interpreting multilingual literature, likely explains the journal editors continued acceptance of such practices (King et al., 2020). Renewed dialogue among authors and editors regarding the essential components of reviews—perhaps an additional objective of this review—could result in a change in publication practices or, at minimum, an explicit commitment to the status quo (Maner, 2014). Likewise, editors may address issues resulting from page limitations through the adoption of online supplements (Tenopir et al., 2011).

Limitations

In highlighting issues with systematic and meta-analytic reviews in behavior analysis, the methods currently employed appear to have perpetuated practices associated with bias. In particular, the present review relied exclusively on peer-reviewed articles obtained from a narrow range of English-language publications. The journal selection process omitted lesser-known or low-impact outlets associated with behavior analysis as well as unpublished work. Difficulty in identifying discrete behavior analysis publications likely resulted in the exclusion of relevant outlets. However, we did not intend to suggest literature reviews, or research of any kind, could achieve perfection. This review assessed article features published in journals broadly representative of behavior analysis (Hantula, Critchfield, & Rasmussen, 2017) to contribute to discussions regarding standards of scholarship in the field (e.g., Petticrew, 2015). As all research, including the current review, inescapably exhibit flaws (Kollins et al., 1999), this analysis eschewed the subtly pejorative term “quality indicator” in order to draw attention to defensible aspects of searches that might nonetheless introduce bias. This study contributes to the field in spite of these shortcomings and remains sufficiently transparent to permit redress in future research.

The coding scheme employed in the present review encompassed methodological transparency (i.e., search details) as well as the implications of prevalent search practices (i.e., the influence of reported criteria findings). Codes did not exhaustively address features examined in previous reviews (e.g., Mackay et al., 2003). The codes applied reflect the focus of this review on features distinct to the research review process (i.e., search and selection procedures). Assessing features such as keywords, research questions, or measurement assessment procedures represents a valuable area of inquiry for future scholarship.

An additional limitation involves the meta-analysis designation, as some studies bearing the term may not appropriately be described as meta-analyses. We intentionally used a liberal definition of meta-analysis in order to allow the identification of articles featuring attempts at quantitative synthesis within a discipline with a tradition of hostility toward statistical analysis (e.g., Kollins et al., 1997). This definition likely inflated the number of identified meta-analyses, in particular in ABAI and SEAB journals. Issues related to the quality of meta-analyses within behavior analysis, and the extent such work conforms with guidelines for quantitative synthesis (e.g., Maggin et al., 2011a, 2011b), though beyond the scope of this review, merit further study.

Future Directions

Recognition of the importance of methodology could contribute to changes in both the consumption and production of literature reviews. Given the current results, consumers should contextualize review findings based on the transparency of methods, the breadth of the search, and the inclusion of multiple sources. Explicitly stating research objectives, search procedures, and selection criteria could improve the utility of reviews more generally. Recent systematic reviews in behavior analysis provide excellent examples that could serve as a model for future reviewers. For example, Sham and Smith (2014) provided a clear description of procedures and included a variety of published and unpublished sources. Miller and Lee (2013) explicitly defined inclusion criteria and assessed the impact of various effect metrics using three separate effect sizes. Authors should include easily obtained unpublished material (e.g., dissertations) whenever possible, especially given (1) the feasibility of small-scale SCD studies and (2) the demonstrable impact of such work on the results of reviews (Ferguson & Brannick, 2012). Despite their potential to mitigate the influence of studies with questionable methodology, reviewers should exercise caution in selecting quality indicators. Current quality indices, such as those of the WWC (2017), omit concerns regarding important aspects of the literature (e.g., graphical display; Kubina, Kostewicz, Brennan, & King, 2017) or include problematic criteria (e.g., relaxed standards for studies involving harmful behavior; Cook et al., 2015).

Aside from referencing models of transparency, we hesitate to advocate for the use of distinct approaches, such as specific set of quality indicators. Because findings may indeed vary with the rigor of experimental procedures, employing quality assessment procedures addresses threats to validity posed by conflating studies regardless of rigor. Standards proliferated by a specific organization—though potentially convenient—need not be used provided authors explicitly describe the guidelines and procedures related to quality assessment.

Likewise, our finding regarding the minimal number of meta-analyses does not necessarily represent an endorsement of a specific form of research synthesis in behavior analysis. The field has indeed avoided issues associated with methods that misrepresent findings through the staunch adherence to scientific principles established by Skinner (1938), Sidman (1960), and others. In more recent studies, scholars have suggested many popular SCD effect sizes misrepresent findings and should therefore be accompanied by visual analysis (e.g., Wolery et al., 2010; Tarlow, 2017). As evidenced by the focus of this review, analysis represents a critical, yet secondary concern. Any analysis must proceed from a defensible process of identifying and selecting evidence, because interpreting unrepresentative literature samples will always produce misleading conclusions. Finally, literature reviews that contribute to assessing the extent of research replicability and generality, rightly observed as guiding principles of behavior analysis, may best involve systematic methods but need not necessarily involve quantitative synthesis. Nonetheless, behavior analysis would clearly benefit with attempting to engage with, constructively critique, and develop behaviorally aligned alternatives for quantifying, comparing, and summarizing behavioral research (e.g., Kollins et al., 1997; Lanovaz & Rapp, 2016; Killeen, 2019). The increasing willingness of behavior analysis journals to confront the challenge of summarizing behavioral research represents a positive step for the field and science more generally (e.g., Critchfield et al., 2000; Lanovaz & Rapp, 2016; Killeen, 2019).

The issues observed in many of the systematic and meta-analyses published in behavior analysis present a number of opportunities for researchers. The growing number of systematic reviews suggests the discipline has begun to favor more transparent, structured approaches. Although current methods may allow for some sources of bias, continued effort to address these issues may result in a body of work that fully conveys the strength of evidence supporting behavior analysis in an objective manner befitting the science. Replicating existing reviews with opaque method sections or limited selection criteria could result in improved guidance for professionals. In addition, such work will allow a more precise, subject-specific evaluation of the impact of review procedures on findings. The apparent dearth of systematic reviews in subdisciplines of behavior analysis requires further scrutiny. However, these fields may benefit from a renewed interest in comprehensive examinations of empirical work.

Confusion surrounding the term “meta-analysis” as used in the field also warrants a more thorough evaluation of existing meta-analyses in behavior analysis and agreement upon what meta-analysis entails (Baron & Derenne, 2000). As with statistical methods in general, adopting a common understanding of a meta-analysis would likely address some of the trepidation regarding its use or misuse in behavior analysis. Many articles described as meta-analyses, and particularly those featuring SCDs, aggregate studies using simple averages or are otherwise not consistent with meta-analyses used in other disciplines (Maggin et al., 2011). Methods of synthesizing SCD consistent with the Cochrane definition have recently proliferated and, given the trend observed in this review, may potentially appear more frequently in behavior analysis journals.

Publication trends suggest systematic reviews may continue to represent the dominant form of literature review in many journals of behavior analysis. Although this represents a positive development, the findings of the current study indicate the limited disclosure of search and selection procedures diminishes the utility of otherwise methodical literature reviews. Literature reviews contribute to claims regarding the replicability and generality of behavioral principles (Sidman, 1960). A greater acknowledgment of the importance of transparent, thorough reviews may enhance applied activities as well as refinements in the theory of behavior analysis.