Table 3.
Research Design Standards and Guidelines
| What Works Clearinghouse | APA Division 12 Task Force on Psychological Interventions | APA Division 16 Task Force on Evidence-Based Interventions in School Psychology | National Reading Panel | The Single-Case Experimental Design Scale (Tate et al., 2008) | Ecological Momentary Assessment (Stone & Shiffman, 2002) | |
|---|---|---|---|---|---|---|
| 1. Experimental manipulation (independent variable; IV) | The independent variable (i.e., the intervention) must be systematically manipulated as determined by the researcher | Need a well-defined and replicable intervention for a specific disorder, problem behavior, or condition | Specified intervention according to the Institute of Medicine’s (1994) classification system | Specified intervention | Scale was designed to assess the quality of interventions; thus, an intervention is required | Manipulation in EMA is concerned with the sampling procedure of the study (see Measurement and Assessment table for more information) |
| 2. Research designs | ||||||
| General guidelines | At least 3 attempts to demonstrate an effect at 3 different points in time or with 3 different phase repetitions | Many research designs are acceptable beyond those mentioned | The stage of the intervention program must be specified (see Rossi & Freeman, 1993) | The design allows for the examination of cause and effect to demonstrate efficacy | EMA is almost entirely concerned with measurement of variables of interest; thus, the design of the study is determined solely by the research question(s) | |
| Reversal (e.g., ABAB) | Minimum of 4 A and B phases | (Mentioned as acceptable. See Analysis table for specific guidelines) | Mentioned as acceptable | N/A | Mentioned as acceptable | N/A |
| Multiple baseline/combined series | At least 3 baseline conditions | At least 3 different, successive subjects | Both within and between subjects Considered the strongest because replication occurs across individuals |
Single-subject or aggregated subjects | Mentioned as acceptable | N/A |
| Alternating treatment | At least 3 alternating treatments compared with a baseline condition or two alternating treatments compared with each other | N/A | Mentioned as acceptable | N/A | Mentioned as acceptable | N/A |
| Simultaneous treatment | Same as for alternating treatment designs | N/A | Mentioned as acceptable | N/A | Mentioned as acceptable | N/A |
| Changing/shifting criterion | At least 3 different criteria | N/A | N/A | N/A | N/A | N/A |
| Mixed designs | N/A | N/A | Mentioned as acceptable | N/A | N/A | N/A |
| Quasi-experimental | N/A | N/A | N/A | Mentioned as acceptable | N/A | |
| 3. Baseline (see also Measurement and Assessment Standards) | Minimum of 3 data points | Minimum of 3 data points | Minimum of 3 data points, although more observations are preferred | No minimum specified | No minimum (“sufficient sampling of behavior occurred pretreatment”) | N/A |
| 4. Randomization specifications provided | N/A | N/A | Yes | Yes | N/A | N/A |