Tactics to Evaluate the Evidence Base of a Nonbehavioral Intervention in an Expanded Consumer Area

Abstract

The purpose of this article is to provide tactics for the behavior analyst to effectively evaluate the evidence base for an unfamiliar nonbehavioral intervention when expanding services. Just as behavior analysts must be aware of fad treatments in autism likely to be encountered, so, too, should practitioners become familiar with potential fad treatments in any expanded area of practice. The present article extends previous work by considering challenges surrounding contact with nonbehavioral literature in the context of an expanded consumer base. The article also considers ethical interactions with nonbehavioral professionals following appraisal of the intervention, and how adopting the strategies listed here can aid in establishing oneself as a resource. Associated barriers and solutions are presented around four tactics: (a) searching the literature, (b) recognizing and evaluating the common properties of fad treatments, (c) distinguishing quality of evidence, and (d) ascertaining behavioral mechanisms of action. Examples from gerontology will be provided to illustrate the use of the proposed tactics.

As behavior analysis continues to experience growth in the area of autism and an increase in public visibility, stakeholders may also see value in behavior-analytic services applied to other domains in addition to autism (e.g., traumatic brain injury, sports, gerontology). Partly as a result of increased visibility in the public eye, several states are passing licensure laws to legally define the scope of practice of behavior analysts, and the language in many of the license bills are open to areas of practice outside of autism (Association of Professional Behavior Analysts, n.d.). LeBlanc, Heinicke, and Baker (2012) suggest that along with the possibility of producing socially important outcomes with underserved populations, applying the technology of behavior analysis to expanded areas can promote greater flexibility in a practitioner’s repertoire. Diversifying one’s consumer base can therefore result in improved financial stability and job marketability as the practitioner successfully navigates a variety of settings and populations. When expanding consumer bases, however, Board Certified Behavior Analysts (BCBAs; hereafter referred to as behavior analysts) need to ensure they are promoting and using evidence-based approaches.

Evidence-based practice (EBP) has been written about in behavior analysis (Slocum et al., 2014; Smith, 2013) and discussed as a clinical process in which the practitioner incorporates the best available evidence, the consumer’s context and preferences, and the practitioner’s own expertise in formulating a treatment plan (Spring, 2007). The particular actions involved in clinical decision-making around the three components of EBP are less defined, and applying EBP when branching out to an expanded area of work in behavior analysis is further unclear. Recently, however, several behavior analysts have made attempts to clarify how to maintain or increase expertise and how to incorporate the best available evidence in service delivery to better guide the practitioner (Brodhead, 2015; Carr & Briggs, 2010; Geiger, Carr, & LeBlanc, 2010; LeBlanc, Hagopian, Maglieri, & Poling, 2002; LeBlanc et al., 2012; LeBlanc, Raetz, Sellers, & Carr, 2015).

Carr and Briggs (2010) and LeBlanc et al. (2012) have written on appropriate scientific databases (e.g., PsycINFO) in which to conduct searches and to stay current with research to contact the best available evidence. As Carr and Briggs (2010) note, several barriers exist to maintaining up-to-date resources, including limited access to scientific databases outside of a university setting, expensive journal subscriptions, and having many journals to follow. The solutions as described by the authors to address such barriers are skills that any behavior analyst should have. LeBlanc et al. (2012) also offer keyword entry suggestions in traumatic brain injury and behavioral gerontology. As they indicate, this is just a first step in contacting the literature. Practitioners would also benefit from specific tactics that inform the reader on how to input keywords and other search terms to obtain the most representative literature sampling, given that PsycINFO operates differently from the Google search engine.

Carr and Briggs (2010) assume that the practitioner already has an established set of problem-solving skills with respect to literature searches. Further, the authors indicate that the increased response effort associated with learning how to efficiently interact with PsycINFO may serve as a barrier to its adoption. A set of search strategies for use with PsycINFO and related scientific search engines might establish greater fluency, which in turn may reduce response effort. The search tactics could then facilitate maintaining competency in relevant practices (code 1.03 in the BACB Professional and Ethical Compliance Code for Behavior Analysts [PECC]; Behavior Analyst Certification Board [BACB], 2014) and establishing competence in applying behavior-analytic practice to expanded populations.

In terms of the clinical expertise component within the EBP model, increasing competence in delivering behavior-analytic services to an expanded population is one part of diversifying one’s consumer base. Once a practitioner has then successfully gained employment in the expanded setting, he or she should anticipate encountering unique ethical and professional issues. One probable situation is having professionals already entrenched in the area conducting nonbehavioral1 interventions common with the population. PECC 2.09d asserts that “behavior analysts review and appraise the effects of any treatments about which they are aware that might impact the goals of the behavior-change program, and their possible impact on the behavior-change program, to the extent possible” (BACB, 2014, p. 9). It is therefore the ethical obligation of practitioners to familiarize themselves with the interventions other professionals in the setting employ that could influence treatment success when implementing behavior-analytic services.

To this end, Brodhead (2015) proposed the use of a problem-solving model when presented with a nonbehavioral intervention in an interdisciplinary team providing autism spectrum disorder (ASD) services. Brodhead’s model included a series of assumptions about the context in which this model would be used. His four assumptions were (a) “that a nonbehavioral colleague has proposed a nonbehavioral treatment” (p. 72); (b) “the BCBA should be adequately trained, or under adequate supervision, to provide services in an interdisciplinary team” (p. 72); (c) “the BCBA has a role in the interdisciplinary team that affords them the opportunity to evaluate and possibly comment on nonbehavioral treatment” (p. 72); and (d) “consent has been provided for the nonbehavioral treatment the BCBA is analyzing” (p. 72). Brodhead further held a focus on how to maintain positive relationships with nonbehavioral colleagues and provided a model for avoiding challenging other professionals until (a) after the behavior analyst has exhausted efforts to understand, research, and then reconcile the nonbehavioral intervention and (b) the “impacts to the client [are] sufficient to justify the possibility of compromising the professional relationship” (p. 73).2 In the step to become familiar with nonbehavioral interventions, Brodhead suggests contacting the literature as a first course of action (p. 73). However, once the behavior analyst gathers relevant studies, additional tactics are still needed to evaluate the quality of that evidence base.

Nonbehavioral interventions, including fad treatments in the area of ASD and intellectual and developmental disabilities (IDD), have been investigated by behavior analysts (e.g., Foxx & Mulick, 2015; Quigley, Peterson, Frieder, & Peterson, 2011). Authoritative texts written for behavior-analytic audiences exist on fad treatments in ASD and IDD, such as Foxx and Mulick (2015), but no well-known book on pseudoscience surrounding other populations, aimed at behavior-analytic audiences, has been published. Additionally, one should consider authoritative reviews pertaining to the expanded consumer area (e.g., Cochrane reports), but the criteria used in the review to evaluate the evidence must also be considered. Some authoritative reviews, for example, may discount the value of single-subject methodologies and therefore might not be the most representative or equivalent appraisal as compared to Foxx and Mulick’s text.

We suppose that outside of one’s area of expertise and literature base in ASD, the detection of and information gathering surrounding pseudoscientific nonbehavioral interventions may not generalize to the expanded area of practice. The purpose of this article is to provide tactics for the behavior analyst to effectively evaluate the evidence base for an unfamiliar intervention when providing services to an expanded population. Given that behavior analysts must be aware of nonbehavioral treatments in autism likely to be encountered (BACB, 2016), so, too, should practitioners become familiar with potential fad treatments in any expanded area of practice. The present article extends previous work on strategies to maintain competency in current behavior-analytic research and practice in the field (Carr & Briggs, 2010) by considering challenges uniquely related to contact with nonbehavioral literature in the context of an expanded consumer base (Leblanc et al., 2012). The article also considers ethical interactions with nonbehavioral professionals in the expanded area following appraisal of the intervention, and how adopting the strategies listed here can aid in establishing oneself as a resource. Associated barriers and solutions are presented around the four tactics3 of (a) searching the literature, (b) recognizing and evaluating the common properties of fad treatments, (c) distinguishing quality of evidence, and (d) ascertaining behavioral mechanisms of action. Examples from gerontology will be provided to illustrate the use of the proposed tactics. Refer to Table 1 for a summary of some barriers and proposed solutions around these four tactics.

Table 1.

Tactics, Barriers, and Solutions to Evaluate Nonbehavioral Interventions in an Expanded Consumer Area

Tactic	Barrier	Solution
Searching the literature	• Unknown search engines and premiere journals that produce the most relevant hits in a specialty area • Paywalls • Lack of training in search engine features	• Identify database search engines. • Use Boolean indicators. • Use forward and backward searching.
Recognizing the common properties of fad treatments	• Pseudoscientific treatments that may have a different appearance outside of ASD/IDD	• Be familiar with common properties of fad treatments (see Table 2).
Distinguishing the quality of evidence	• Lack of graduate training on group design methodology • Lack of graduate training on group design strengths and weaknesses	• Be familiar with common research methodologies. • Distinguish limited generality or overextensions of treatment. • Evaluate the correspondence of face-value descriptions to the actual evidence base.
Ascertaining the behavioral mechanism of action	• Unknown behavioral processes promoting behavior change	• Review procedures. • Analyze molecular contingencies in a given protocol.

Tactic: Searching the Literature

Identify Database Search Engines

The particular search engine the behavior analyst uses influences the type and number of articles found and is therefore a critical component of the information-gathering process. The behavior analyst should expect barriers associated with locating useful databases outside of traditional behavior-analytic outlets, including unknown search engines and relevant premiere journals, paywalls, and a lack of formal training in search engine features. Here, we summarize solutions to these barriers for maximizing contact with the evidence base as the practitioner gathers information.

When first exposed to an unfamiliar intervention, a behavior analyst should conduct a literature search to obtain information on prior evaluations of the practice. As a function of past training, the behavior analyst may first use the resources readily available and with which he or she is most familiar. For example, the BACB offers access to the behavioral journals Journal of Applied Behavior Analysis, Journal of the Experimental Analysis of Behavior, and Behavioral Interventions for BCBAs and Board Certified Assistant Behavior Analysts (BACB, 2016). We recommend behavioral searches as the first step, but relying solely on this method can be problematic.

If a search in familiar behavioral journals does not produce relevant empirical investigations, then we recommend expanding the search by using resources that pull from a pool of behavior-analytic and non-behavior-analytic journals, such as Google Scholar and PsycINFO (refer to Table 1 in Carr & Briggs, 2010, for a list of behavior-analytic journals that are indexed by PsycINFO). Leblanc et al. (2012) also suggest using “national organization resource websites to search for handbooks, manuals, or guides” (p. 8) to address barriers in locating search databases outside of behavior analysis of which the behavior analyst may not be aware. Furthermore, websites with search database features such as ResearchGate provide preprints that can also be useful if paywalls to journal articles serve as a barrier. PsyArXiv also provides digital object identifiers and is indexed by Google Scholar.

Burgio and Kowalkowski (2011) suggest that searches for the evidence base in behavioral gerontology research should include journals outside of behavior analysis, such as Clinical Gerontologist and other aging outlets, perhaps because behavior analysts conducting research with older adults will likely disseminate the evidence to the audience specific to that area. Restricting the search to include only behavioral journals may therefore not produce relevant hits. The behavior analyst must look to the broader gerontology literature to contact the evidence base for further examination.

As an example, if a behavior analyst beginning to provide services to older adults in an assisted-living facility encounters staff using bright light therapy (a treatment aimed to alter circadian rhythms and in turn believed to impact agitation related to sundowning), he or she will want to evaluate the evidence base for that intervention. A search for “bright light therapy” in behavioral journals such as Behavior Modification will produce zero relevant search hits.

More medically based search engines may be useful given the particular subject matter or professional setting in which the behavior analyst is conducting professional work, including PubMed and Web of Science. Nursing journals that frequently publish work with the consumer base, for example, may have published evaluations on the relevant treatment. PubMed may not have a database that is as robust in psychological literature as PsycINFO, limiting its utility in finding the evidence base for any nonmedical intervention, but PubMed should be considered because it (a) does not have the Boolean problem when constructing search parameters (see the next section) and (b) has all National Institutes of Health–sponsored research available for free. Throughout this article, PsycINFO will be the primary search engine used for illustration purposes, but we encourage the use of other science databases because each search engine has associated pros and cons. In the following sections, we describe a few features that can impact the success of the behavior analyst’s search.

Boolean Indicators

When using PsycINFO, terms must be carefully entered to obtain the best results. Entering all terms into the basic search field, as is common with Google, will overly restrict the search parameters because PsycINFO searches for articles that only contain all of the included terms. Given the relatively high number of new behavior analysts in the field who may be more accustomed to using Google, it is quite possible that behavior analysts might not be familiar with Boolean indicators. For example, when a behavior analyst begins the literature search for reality orientation (a therapy in which reminders on date, time, and location are provided to an older adult), typing the terms “reality orientation dementia aggression” (a common approach in an internet search) in PsycINFO produces one report of caregivers’ strategies used when wandering is observed. We therefore recommend the use of Boolean indicators to improve search results when using science search engines (Calhoun, 2013). To do so, first select “advanced search.” Below the main search bar, you will see additional fields specified by the Boolean indicators. Entering a term in the “and” field will result in hits that contain only both terms, thereby restricting your search results. Adding a term to the “or” field will produce results that contain either term, thereby expanding your results. Check boxes for “peer-reviewed” and “scholarly journals” can also be selected.

Backward and Forward Searching

Following an initial search, PsycINFO, Web of Science, and PubMed allow for backward searching by clicking on the “references” text within any search hit or by referring to the reference section of the article. Backward searching provides information on previous literature by the authors of the work, as well as other authors’ research activities as the lines of research developed.

Forward searching allows for the identification of research that has been published after the original article in question. In PsycINFO, forward searching can be done by clicking the “cited by” text for that article in the search results. Searching in this way allows for contact with follow-up studies.

We suggest that the behavior analyst terminates the literature search when at least several empirical studies (i.e., those that operationally define the independent and dependent variables and systematically collect data) have been found across research groups (i.e., studies out of a particular laboratory or collaborative group) and multiple search engines have provided cross-referenced search hits (i.e., the same article is found across more than one search engine). Having several studies improves the extent to which the behavior analyst can then effectively evaluate the evidence base.

Tactic: Recognizing the Common Properties of Fad Treatments

After the behavior analyst has gathered relevant search results, it is important to attend to the inclusion of any language observed in (a) the obtained empirical studies, (b) Google search hits, and (c) how professionals in the consumer area have talked about the intervention that is typical of fad treatments in other areas (e.g., sensory diets with ASD). An associated barrier with this tactic is that fad treatments in the expanded consumer area may take on a different form, thereby reducing detection. In addition, an important point to consider as one evaluates a nonbehavioral treatment is that any one identified characteristic it possibly shares with other known fad treatments is not sufficient to write it off as unvalidated. Rather, identifying the common properties of fad treatments allows for a first step toward skeptical appraisal to be followed up with a more rigorous evaluation. See Table 2 for a list of these common properties from Vyse (2015), with examples in how such language could be used in the area of aging. We refer the reader to Foxx and Mulick’s (2015) edited book Controversial Therapies for Autism and Intellectual Disabilities: Fad, Fashion, and Science in Professional Practice (2nd ed.) for a detailed discussion around pseudoscientific practices. Although the analyses in the book are primarily geared toward treatments in the area of special education, the conditions under which fad treatments emerge, along with their defining features, are relevant to the present article. We also include discussion of a selected nonbehavioral treatment used in aging settings under this tactic because it exemplifies several of the properties in common with other fad treatments.

Table 2.

Common Properties of Fad Treatments

Property	Example
Cures	Miracle cure for Alzheimer’s disease
Important-sounding but vague benefits	Improvements in memory (without describing the measurement system)
Major gains that cannot be studied	Sensory processing
Uncontrolled studies	Referring to research that does not have controlled methods
Sophisticated technology used in unvalidated ways	Measuring sleep as evidence for improvements in wandering
Opposition from the “establishment”	Censorship from big pharma on a discovered cure for memory loss
Criticisms of validated treatments	Negative side effects of a medical treatment used as evidence for the effectiveness of an unvalidated nutritional supplement
Subjective evidence	YouTube videos with older adults in therapy
Hypothesis of a core deficit in sensorimotor function	Snoezelen room and sensory processing in neurocognitive disorders (NCD)
Natural intervention	Bright light therapy and natural benefits of sun exposure

Adapted from Vyse (2015)

Be Familiar with Common Properties of Fad Treatments

ASD and IDD are replete with fad treatments in part because first-line treatments do not bring about complete reversal of the behavioral symptoms that originally led to a diagnosis (Vyse, 2015). A parallel exists for neurocognitive disorders (NCD)—formerly referred to as dementia—in that no treatment is currently available that reverses the condition. We recommend cautious appraisal when working in expanded areas in which the current first-line treatments are burdensome to consumers, families, and staff, as fad treatments may be an appealing alternative to procedures viewed as noxious. In specialized memory care units, for example, pharmacological interventions, restrictions in freedom of movement, and other strategies used to manage the behavioral excesses of NCD may be common. Such conditions set the stage for the rapid emergence of fad treatments.

As an example of how the current tactic could be applied in an aging setting, consider bright light therapy. Bright light therapy is a treatment aimed to alter circadian rhythms and involves the use of a light source at some intensity. The light source can take the form of a light box (Fetveit & Bjorvatn, 2005) or ambient lighting (Sloane et al., 2007) and is delivered to the older adult at specified times. Measures used to assess the effects of bright light therapy on sleep have included accelerometers (Fetveit & Bjorvatn, 2004), serum melatonin levels, and sleep diaries recorded by nursing staff (Mishima et al., 1994).

The use of technology either as part of a treatment or to measure the effects of a treatment is not a sole indication that the techniques are scientific. Perhaps for some, science is thought of as a collection of lab equipment that includes technologies such as microscopes, electronics, and chemical materials. Although these tools are often used in certain types of research, proponents of questionable treatments may capitalize on this notion of science and insist that the treatment is supported by evidence because it includes the use of technology. Whenever a technology that is unfamiliar to the behavior analyst is used in a nonbehavioral treatment, she or he should engage in research and evaluate the literature base on the validity of the technology.

Validity in this context refers to how well the technology measures the area of interest (Kazdin, 2010). Therefore, it is not the presence or absence of a given technology that is critical but whether the technology is actually measuring what it is claiming to measure. Bright light therapy becomes unvalidated when the clinical goal moves from treating the sleep domain to treating the behavioral domain (see “sophisticated technology used in unvalidated ways” in Table 2). Without additional evidence that suggests its effectiveness, claims that bright light therapy can be used for challenging behaviors associated with NCD should be questioned. Multiple studies have concluded that there are insufficient data that bright light therapy, although perhaps effective for changing circadian rhythms and improving sleep quality, is an effective treatment for agitation and wandering (Burns, Allen, Tomenson, Duignan, & Byrne, 2009; Lyketsos, Veiel, Baker, & Steele, 1999).

Nevertheless, proponents utilize bright light therapy with older adults because they characterize challenging behavior as being a function of disrupted circadian cycles (see “hypothesis of core deficit in sensorimotor function” in Table 2). Barrick et al. (2010), for example, argue that because agitation, sleep disorders, and circadian disruptions are prevalent among older adults, treating the disrupted circadian cycle could presumably decrease challenging behaviors. Although the dysregulated circadian cycle conceptualization makes intuitive sense, it is couched in correlational data and is not conceptually systematic according to behavioral principles. Moreover, when studies measure changes in sleep quality and challenging behavior, they may emphasize improvements in sleep even with no changes in behavior.

The behavior analyst may be tempted to interpret this to mean that circadian disruptions function as a motivating operation for challenging behavior (i.e., sleep deprivation increases sundowning behaviors), but out of the published functional analyses on wandering in older adults, lack of sleep has not yet been found to be a critical variable. Instead, wandering has been shown to be a function of staff attention (Dwyer-Moore & Dixon, 2007) or access to tangible items (Heard & Watson, 1999). Further research that is conceptually systematic, behavioral, analytic, and technological may provide evidence of the role of sleep in challenging behavior as has been shown with other populations (e.g., Kennedy & Meyer, 1996), but to adopt bright light therapy as an approach without the inclusion of a functional analysis moves the behavior analyst from an inductive logical perspective to a deductive logical perspective, which is antithetical to a behavior-analytic approach.

Tactic: Distinguishing the Quality of Evidence

Once the behavior analyst has identified or ruled out any problematic language included in the description of a nonbehavioral treatment, the task then moves to critically examining the strengths and weaknesses of the evidence. A common barrier in this effort relates to a lack of formal training on group design methodology. Solutions for overcoming this barrier when distinguishing the quality of the evidence involve an understanding of research methods common in the area, how generality of the findings are described and supported, and noting the degree to which a discrepancy exists between face-value descriptions of the nonbehavioral treatment and the actual evidence. It is important to note that any cited evidence must be evaluated with the same standards used for studies that have been established as empirically supported. Standards include, but are not limited to, adequate methodology that rules out alternative explanations for changes in dependent measures, operationally defined outcomes, and controls for observer drift and bias. As a way of critically reviewing the evidence put forth by proponents, the behavior analyst should ask questions such as “Is some other variable responsible for improvements?” “Are interobserver agreement data reported?” “Are treatment integrity data collected on treatment implementation?” and “Are experimenters or data collectors blind to conditions?”

Be Familiar with Common Research Methodologies

As a behavior analyst, expertise in single-subject design methodology is expected, given that such training is included in the BACB task list (BACB, 2012, 2017). Other research methods, however, including group designs, are not on the BACB task list. Without an explicit course on non-behavior-analytic research methods, it is possible that the behavior analyst may not receive any formal training in a variety of designs as they progress through a behavior-analytic graduate program.

Brodhead (2015) has recognized the value of research designs used by other disciplines in order to answer certain experimental questions that would increase knowledge on ASD treatment. We also urge acknowledgment of designs not traditionally employed in behavior-analytic research in the expanded consumer area, as an increased understanding of the methodology improves the extent to which the behavior analyst can evaluate the research on its own terms. Gaining familiarity allows the behavior analyst to speak to the particular ways the design addresses problems with internal and external validity without resorting to the generic concerns behavior analysis raises with group designs.

Information regarding research methods in the expanded consumer area is most likely to be found in specialized textbooks rather than in journal articles. The behavior analyst should adjust search parameters described in the tactic “searching the literature” to produce relevant hits for book chapters and textbooks. If using an online library database, some resources include full electronic copies of books. When seeking out research methods used with older adults, suggested search terms could include “aging,” “older adults,” “clinical,” “research,” and “methods.”

Common research designs used with aging populations include cross-sectional and longitudinal designs. Examples of textbooks that provide information on these designs include ones that offer a general overview of the population (e.g., Moody & Sasser, 2017; Morgan & Kunkel, 2015) or edited books on clinical research methodology (e.g., Thomas & Herson, 2011). The specific procedures of the respective designs along with their strengths and weaknesses are beyond the scope of this article, but the reader is referred to Feliciano, Yochim, Steers, Jay, and Segal (2011) for further details on these common designs used with older adults and more general issues to consider when conducting research with this population.

Distinguish Limited Generality or Overextensions of Treatment

After evaluating the quality of research designs used in studies on the nonbehavioral treatment in question, the next step is to review statements of generality and extensions of the treatment. This information usually can be found in the Discussion section of empirical reports and should be considered regardless of whether the quality of the research design is high (i.e., good internal validity does not necessarily equal good external validity). Identifying potential issues in generality relates to whether or not the procedures as reported would be effective in different settings, with different individuals running the protocols, or with different populations. Questions that should guide the behavior analyst’s appraisal include “Do the results obtained support the conclusions the author makes?” “Could similar results be obtained if changes to the intervention are made, or do the procedures have to be delivered exactly as written?” and “Would the reported procedures be appropriate with a different population, setting, target behavior, or behavioral function?”

Consider the Snoezelen room, a type of multisensory environment. Proponents argue that Snoezelen rooms have been researched with a variety of populations, such as individuals with traumatic brain injury, ASD, and pregnant women suffering from depression and anxiety (Snoezelen Multi-Sensory Environments, n.d.), but relying on any evidence not done with older adults is insufficient to suggest that Snoezelen rooms would be effective for an older adult population. It does not follow that because an intervention is effective with one population, that it will be equally effective when applied to another population without special considerations of that population.

In behavioral gerontology, it would be a similar error to assume that the research on procedures used in applied behavior analysis with ASD and IDD populations is also sufficient evidence that those same procedures would be effective for older adults with NCD. Behavioral gerontologists have spent much effort conducting research to determine how well behavior-analytic assessment and intervention procedures translate and what, if any, alterations to those procedures should be made when working with older adults. Stimulus preference assessments, for example, typically demonstrate that edibles overpower the selection of tangible items when presented together for individuals with ASD. Virués-Ortega, Iwata, Nogales-González, and Frades (2012) demonstrated that with older adults under stimulus preference arrangements, tangible items overpowered edibles, standing in contrast to previous findings with other populations. The researchers found that preference assessments were effective in identifying preferred events to be used as reinforcers in a later task, illustrating how procedures that are well established with one population need to be evaluated with the population of interest to ensure that the effects carry over. Great care is taken in behavior analysis to conduct replications to directly avoid overgeneralizing and inappropriately extending treatments.

Evaluate the Correspondence of Face-Value Descriptions to the Actual Evidence Base

Once the behavior analyst has critically examined both the design of studies used on the nonbehavioral treatment and any limiting factors relating to the generality of findings, it may be valuable to cross-check how often descriptions of the treatment in question contain positive or negative judgments (e.g., whether it is “good,” “bad,” or “dangerous”). Doing so will help prepare the behavior analyst for discussion on the nonbehavioral treatment that may come up in an interdisciplinary meeting in the new consumer area. This can be done by reviewing the top hits in Google or Google Scholar.

When searching Google with the terms “bright light therapy dementia,” a top hit at the time of this writing frames bright light therapy as a complementary intervention to medication and one that improves sleep cycles, decreases wandering, and improves cognition and behavioral functioning (Heerema, 2017). Proponents with which the behavior analyst interacts in the expanded consumer area may use similar language that casts the therapy in a positive light. As a final example, doing a Google search on reality orientation produces a few hits that describe the adverse reactions to reality orientation, including increases in agitation, and therefore cast reality orientation in a more negative light. Knowledge of the positive or negative attributes made toward a nonbehavioral intervention that are easily accessible to the public allows the behavior analyst to better anticipate and advocate for empirically supported approaches.

Tactic: Ascertaining the Behavioral Mechanism of Action

This tactic relates to PECC 4.01, conceptual consistency, and 6.01, affirming principles (BACB, 2014). A barrier associated with this tactic is that the behavioral processes possibly underlying the procedures are not known. If aspects of the treatment are considered to have an effect based on a possible behavioral function, then the conditions under which the nonbehavioral treatment may be effective can be isolated and manipulated for further analysis. The behavior analyst seeks to answer the following questions: “Exactly what is being presented or removed in the consumer’s environment under this treatment?” and “Is treatment success possible when it is translated into behavioral principles?” (Brodhead, 2015, p. 74). Two strategies to carry out an evaluation couched in behavior-analytic terms, discussed next, are reviewing the replicability of the treatment’s procedures and analyzing the molecular contingencies in the protocol.

Review Procedures

Found in the Method sections of studies, reports of procedures allow the behavior analyst to gauge how technological (i.e., reproducible) the treatment is (cf. Baer et al., 1968). When descriptions of the treatment are sufficiently detailed, replicating the procedures is easier. Important aspects of the procedures to initially focus on include how the environment is set up prior to the treatment session, what is presented or removed in relation to the consumer’s behavior, how behavior is measured, and the manner in which any other type of interaction is structured.

Upon review of reality orientation, large variations on procedures exist in the literature (Spector, Davis, Woods, & Orrell, 2000) and often descriptions of procedures are brief. The location of reality orientation stimuli (e.g., times, dates), the format of the presentation (e.g., the size and font of the text, the colors used on the text board), and how staff use the reality orientation stimuli with older adults (e.g., do staff point at the board while stating the date, time, and an upcoming activity, or do they only present the information verbally, outside of view of reality orientation materials, and do staff continue to re-present the information if the resident disagrees or argues?) are not described. Reality orientation’s replicability is therefore hindered by poor descriptions of procedures, limiting the behavior analyst’s confidence in being able to recreate the treatment as arranged by reality orientation researchers for further evaluation.

Bright light therapy has better replicability because the procedures are sufficiently detailed, which allows the behavior analyst to recreate the intervention. Lyketsos et al. (1999) state the intensity of light in scientific units (i.e., lumens) and the distance from the stationary light box to the participant, describe instructions provided to the participants, and note that a nursing staff member was present in the room for the entire session. In addition, the authors indicate that activities such as watching television or listening to music were freely available throughout sessions. Burns et al. (2009) further report that the supervising staff member engaged the participant in conversation and also distracted participants who tried to leave the session room. For bright light therapy interventions that alter ambient lighting instead of using a stationary light box, Sloane et al. (2005) described not only that light fixtures in public areas of the long-term care facility were altered but also that window coverings were modified for more daylight to enter rooms, ceilings and walls were repainted, and floors were carpeted. Thus, the precise physical manipulations done to the area above and beyond changing room lights are well understood by the reader. Knowledge of precise manipulations allows not only for replication but also for analysis of the possible behavior-environment relations underlying the procedures when it is understood how stimuli are presented, withdrawn, or withheld surrounding behavior.

Analyze Molecular Contingencies in a Given Protocol

Knowing what is done in the procedures helps the behavior analyst understand how treatment is structured. By identifying the behavior-environment relations in a nonbehavioral treatment, the behavior analyst identifies possible functional units embedded within the procedures, and this helps the behavior analyst understand why behavioral effects under such conditions might occur. Given the procedures described with bright light therapy, it is possible that improvements in target behavior such as wandering may be due to access to activities and one-to-one staff attention rather than exposure to the light box. If we consider the function of the participant’s behavior to be positive reinforcement in the form of tangibles and/or attention, bright light therapy sessions could be conceptualized as a fixed-time schedule of reinforcement, resulting in a decrease in wandering. Improvement in behavior is therefore artifactual in relation to bright light therapy despite proponents stating otherwise. Analyzing the behavior-environment relations provides the behavior analyst with a more practical view of the procedures that are in line with behavior-analytic interventions, allowing for further environmental manipulations that confirm or reject the conceptualization.

An experimental demonstration similar to Mason and Iwata’s (1990) analysis could be extended to other sensory-based treatments, including the Snoezelen room used with older adults with NCD. Mason and Iwata showed that improvement seen in behavior during sensory integration (SI) sessions were attributed to the operative behavior-environment relations and not a sensory diet because the investigators arranged adequate controls that isolated procedural elements of SI. With the Snoezelen room, the absence of aversive stimuli such as staff demands (i.e., being nondirective), the presence of items and activities, and staff attention are potential functional units of the procedure that could have an effect on an older adult’s behavior that is sensitive to those environmental events. Without isolating those aspects, however, proponents can still argue for an alternative explanation based in SI. Maseda et al. (2014), for example, conducted a single-stimulus preference assessment and incorporated preferred sensory-based items and activities into the Snoezelen room. Regardless of outcomes, the study did not control for the possibility that access to preferred items promoted engagement that was incompatible with challenging behavior regardless of what room they were in. The behavior analyst moving forward could consider teasing out this variable with a functional assessment to provide evidence about the behavioral mechanism of action.

After reviewing procedures and considering the contingencies of reinforcement in the nonbehavioral treatment, the behavior analyst is in a stronger position to consider whether the treatment is contraindicated (e.g., is treatment reinforcing challenging behavior?). Brodhead (2015) notes that when client safety is at risk, “it is recommended that he or she [the behavior analyst] addresses the proposed treatment with the nonbehavioral colleague” (p. 74). Brodhead suggests the behavior analyst should also assess the possibility that the nonbehavioral treatment interferes with consumer goals if treatment appears to be contraindicated. Although outside the scope of this article, the reader is referred to Brodhead for additional decision-making steps to maintain professional relationships with nonbehavioral proponents following an analysis of the behavior-environment relations within that treatment.

As a final point of consideration, it is important to know how the treatment is being implemented in the setting regardless of the extent to which it has been identified as empirically supported (or not supported) after using the tactics discussed previously. How well a procedure is being conducted and the strength of that procedure’s evidence base are two separate matters. Questions surrounding this point that the behavior analyst should ask include the following: “How is the intervention being implemented?” “How is it being measured and evaluated in the natural environment?” “How are staff monitoring consumer progress throughout the intervention?” and “What are the clinical criteria for making changes, and how are changes to the intervention being made?” Such questions allow the behavior analyst to gain insight into how staff are implementing the treatment and to acknowledge if any discrepancy exists between the procedures described in research and the procedures applied in practice.

Summary

In this article, we have discussed four tactics, their related barriers, and solutions for the behavior analyst’s use to evaluate the evidence base of an encountered nonbehavioral treatment when entering an expanded consumer area. We discussed these tactics with assumptions presented by Brodhead (2015) about the context in which these tactics might be appropriate and with the understanding that care must be taken to ensure that the interdisciplinary relationship is not negatively impacted, unless the benefit to the consumer outweighs the potential negative impact. With the examples posed from aging settings, literature searches on reality orientation, bright light therapy, and Snoezelen rooms produced an abundance of hits across Google, Google Scholar, and scientific database search engines. Furthermore, aspects of each nonbehavioral treatment included language that corresponded to the language used with fad treatments. Technically speaking, none of the nonbehavioral treatments reviewed in this article qualified as fads per se because their adoption by nonbehavioral clinicians has persisted across time. Fad treatments, by definition, experience a rapid surge in popularity before quickly losing ground (Vyse, 2015). Nevertheless, the inclusion of such language within the reviewed nonbehavioral treatments prompted skepticism.

Distinguishing the quality of evidence found from the literature searches further supported skepticism around the questionable language used to describe and promote the nonbehavioral treatments. Bright light therapy has some supporting evidence toward its effectiveness in regulating sleep cycles, but little evidence that the treatment is effective in decreasing challenging behavior such as wandering. Moreover, despite claims to the contrary made by proponents of Snoezelen rooms, well-controlled studies found little convincing evidence that the multisensory intervention decreased challenging behaviors. If one also considers possible behavior-environment relations embedded in the procedures, any beneficial effects can be viewed as a function of measurable environmental events rather than a hypothesized internal sensory deficit. Finally, peer-reviewed publications on reality orientation did not include sufficient procedural details to consider possible behavioral mechanisms of action, but bright light therapy and Snoezelen room procedures were found to be replicable.

Next Steps

Although an aging-specific setting has been used to illustrate the tactics, we remind the reader that a behavior analyst should engage in the proposed tactics whenever he or she comes across an unfamiliar nonbehavioral treatment. The process of engaging in critical appraisal of an intervention outside of behavior analysis is a complicated one, but behavior analysts are ethically bound to do so. By detailing specific actions the behavior analyst should take, it is our goal that the process becomes more formalized to better guide behavior analysts in clinical decision-making in these circumstances and to promote alignment with EBP.

After the behavior analyst has obtained adequate information through the use of the tactics and is confident in his or her ability to discuss the nonbehavioral treatments with other professionals in the consumer area, the final challenge is to present the findings in such a way that is in the spirit of PECC 2.03b: “When indicated and professionally appropriate, behavior analysts cooperate with other professionals in a manner that is consistent with the philosophical assumptions and principles of behavior analysis, in order to effectively and appropriately serve their clients” (BACB, 2014). Although the proposed tactics in this article allow for the behavior analyst to discuss the nonbehavioral treatment in a manner consistent with the philosophy and principles of the science of behavior, additional steps are necessary to increase the likelihood that the behavior analyst does so in a professionally appropriate way. Brodhead (2015) provides problem-solving steps that can be implemented to this end. The problem-solving steps focus on considering how and when to address concerns over a nonbehavioral treatment in such settings and include special attention to whether client safety is at risk, whether it interferes with client goals (briefly discussed at the end of the tactic on ascertaining the behavioral mechanism of action), and ultimately whether to potentially compromise the quality of a relationship with a professional because the treatment is found to negatively impact the consumer.

Bailey and Burch (2010) discuss ethics in daily life as one of their 25 essential skills for the professional behavior analyst. They remind the behavior analyst that nonbehavioral colleagues are not bound by our standards and offer basic strategies on persuasion and influence, as well as negotiation and lobbying. Although beyond the scope of this article, the reader is directed toward these essential skills of professional behavior analysts, and they may help to ensure that behavior analysts are in a position to collaborate, suggest, and guide, rather than coming across as critical and demanding to nonbehavioral colleagues. It is important to note clients’ lives and access to therapeutic environments are at stake, but helping to guide other professionals toward EBP allows for the opportunity to teach and establish repertoires so that they will seek out such practices later. Additionally, this can help prevent behavior analysts from becoming stimuli that colleagues work to avoid.

By knowing when and how to cooperate with others, behavior analysts situate themselves as a behavioral resource in the expanded consumer area by maintaining positive relationships with key professionals. When the behavior analyst has effectively established him- or herself in this manner, it is likely that individuals in the consumer area will continue to approach the behavior analyst for consultation on other cases or treatments. In doing so, the behavior analyst becomes a part of the decision-making structure in the setting and opens up additional opportunities to expand behavioral services.

After the behavior analyst has demonstrated her or his value to professionals in the setting, knowledge of other aspects that make up the service delivery culture are needed to maintain this position. Important aspects of the culture with which the behavior analyst should be familiar include the nonbehavioral professionals’ treatment orientation, common credentials such as certification and licensure, and training background. In behavioral gerontology, the activity director is one common professional the behavior analyst may encounter in an assisted-living facility. The activity director is typically responsible for managing a variety of one-to-one or group activities in the aging facility. Knowledge of the activity director’s role in the organizational system can facilitate successful collaboration as the behavior analyst works to increase activity attendance and participation with a consumer, for example.

The steps the behavior analyst takes to effectively familiarize him- or herself with an unknown treatment when expanding his or her professional scope of practice helps him or her retain a behavior-analytic identity. This identity is crucial in preventing deferral to nonbehavioral professionals in an interdisciplinary context and has significant benefits for consumers, the behavior analyst, and the field of applied behavior analysis in showing that the principles and procedures of behavior analysis have diverse application.

Compliance with Ethical Standards

Conflict of Interest

Christopher Walmsley declares that he has no conflict of interest. Jonathan C. Baker declares that he has no conflict of interest.

Ethical Approval

This article does not contain any studies with human participants or animals performed by any of the authors.