Table 3.

CEC Quality Indicator Ratings for Reviewed Studies

CEC QI Criterion	Iwanaga et al. [2014]		Pfeiffer, Koenig, Kinnealey, Sheppard, and Henderson [2011]		Schaaf et al. [2014]
	Rating	Comments	Rating	Comments	Rating	Comments
1. Context and setting	The study provides sufficient information regarding the critical features of the context or setting relevant to the review; for example, type of program or classroom, type of school, curriculum, geographic location, setting, etc.
Describes critical features of context or setting relevant to review, including type of program or classroom, type of school/facility, curriculum/intervention geographic location, community setting socioeconomic status, and, physical layout.	Yes	Out‐patient OT program (ASI and group social skills training) provided in specified medical/education center and university clinics in Nagasaki Japan; similar SES by proxy as subjects came from Nagasaki; description of SI clinic with fidelity requirements noted	Yes	Supplemental OT services provided at rural summer camp in Pennsylvania; similar SES by proxy as subjects attended same community‐based camp program; ASI provided in 3 SI rooms that met ASI fidelity requirements and OT‐fine motor group provided in 1 fine motor room	Yes	Out‐patient OT program at children's hospital in a New Jersey community provided ASI; SES reported by proxy via education level (largely college educated); equipment and space met ASI fidelity standards
2. Participants	Provides sufficient information to identify population of participants to which results may be generalized and determine or confirm whether participants demonstrated disability or difficulty of focus.
2.1 Describes participant demographics relevant to the review including characteristics such as gender, age/grade, race/ethnicity, socioeconomic status, language status, etc. 2.2 Describes disability or risk status of the participants and method for determining status.	Yes	2.1 Subjects primarily males, ages 2.9–6.2 years; race/ethnicity presumed Japanese; language status not relevant but IQ scores >70. 2.2 ASD diagnosis per DSM‐IV; information from clinical records and pediatrician diagnosis; sensory integrative disorder identified by JMAP; participant ages appropriate for measures; IQ scores determined using appropriate tool; specified subjects attended regular nursery school or kindergarten, had no previous therapy and parents/teachers, etc. had no previous knowledge of sensory integration	Yes	2.1 Subjects primarily males, ages 6–12 years; language status not relevant but ASD diagnosis per reported DSM‐IV criteria. SES and race/ethnicity not included. 2.2 Autism diagnosis verified by report from qualified diagnosticians; sensory integration deficits identified by SPM and comprehensive evaluation by an expert ASI clinician	Yes	2.1 Subjects primarily males, 4.0–8.11 years, primarily Caucasian; cognitive level reported; language status not relevant. Parent level of education as an estimate of SES. 2.2 ASD diagnosis and autism severity identified by licensed psychologist using ADI‐R and ADOS; sensory integration difficulty identified by scores on SP and SIPT
3. Intervention agent	Provides sufficient information regarding critical features of intervention agent.
3.1 Describes role of intervention agent and relevant background variables. 3.2 Describes any specific training or qualifications required to implement the intervention, and indicates that the interventionist has achieved them.	Yes	3.1 Group therapy provided by occupational therapist, speech therapist and 3 nursery school teachers to small group of 5–6 children; first author (OT) administered JMAP evaluations and SI treatment 3.2 Interventionist OT certified by Japanese Sensory Integration Association; group interventionists backgrounds not stated.	Yes	3.1 Fine motor group provided by OT graduate students, with supervision; ASI intervention provided by OTs; at least one treatment session/child was video recorded and scored for fidelity to ASI supporting appropriate role for interventionist 3.2 Evaluators and interventionists received training before implementing intervention including principles from ASI fidelity measure	Yes	3.1 ASI intervention provided by licensed OTs (n = 3; mean years of experience = 15, range 12–20 years), experienced working with children with ASD; evaluators (blinded to conditions) were also licensed OTs; comparison group was usual care as reported by parents 3.2 Interventionists and evaluators completed certificate programs in ASI including use of the SIPT
4. Description of practice	Provides sufficient information regarding critical features of practice (intervention); practice is clearly understood and can be reasonably duplicated
4.1 Study describes detailed intervention procedures such as instructional behaviors, critical or active elements, manualized or scripted procedures, dosage, intervention agents' actions, or cites one or more accessible sources that provide this information. 4.2 When relevant, study describes materials (e.g., manipulatives, worksheets, timers, cues, toys), or cites one or more accessible sources providing this information.	Yes	4.1 Practice described relative to components of ASI fidelity measure (which was not available at time of this study); dosage 1 hr/week for 8–10 months (average of 9.3 months; no specific manual, but intervention followed 10 key therapeutic strategies of ASI Group therapy dosage 1.5 hr/week for 8–10 months (average 9.3 months); activities described briefly (social skill training, communication training, kinetic activities, child–parent play); environment not described 4.2 Provides description of materials, activities and additional resources for ASI intervention	Yes	4.1 Practice described as SI intervention based on ASI fidelity measure and grounded in Ayres SI treatment theory; dosage 18 sessions, 45 min each for 6 weeks, one child received 17 sessions; intervention followed 10 key therapeutic strategies for ASI Fine motor treatment training and intervention based on fidelity measure developed for this study; Basis for treatment for both groups described and examples provided with literature resources cited 4.2 Specific materials not described but general activities for both groups presented with additional resources provided for ASI intervention	Yes	4.1 Practice well described as ASI; dosage 3 hr/week, 1‐hr sessions, total of 30 sessions over 10–12 weeks; specific details of intervention provided with reference to manual pending publication at time of study; multiple appropriate sources for more information cited; adherence to fidelity conducted in a feasibility study prior to study and results published elsewhere 4.2 Examples and description of some equipment/materials and f specific activities with multiple appropriate sources for more information
5. Implementation Fidelity	Practice is implemented with fidelity.
5.1 The study assesses and reports implementation fidelity related to adherence using direct, reliable measures (e.g., observations using a checklist of critical elements of the practice). 5.2 The study assesses and reports implementation fidelity related to dosage or exposure using direct, reliable measures (e.g., observations or self‐report of the duration, frequency, curriculum coverage of implementation). 5.3 As appropriate, the study assesses and reports implementation fidelity (a) regularly throughout implementation of the intervention, and (b) for each interventionist, each setting, and each participant or other unit of analysis.	No	5.1 Study refers to principles of ASI, but does not use fidelity measure 5.2 Indicates dosage, but does not use a fidelity measure 5.3 N/A since fidelity not measured	Yes	5.1 ASI fidelity measure used; all ASI sessions met fidelity criteria of >80; fine motor fidelity measure developed for study including 3 main focus areas, fidelity criteria, and score range to support assessment of fidelity; all fine motor sessions met fidelity criteria of ≥75 5.2 Dosage indicated by session length, number of sessions/week; total number of sessions; subjects completed 18 sessions, except one subject completed 17 5.3 All sessions checked for fidelity	Yes	5.1 Manualized intervention with adherence to fidelity; reported psychometrics for fidelity measure; all intervention sessions recorded; 10% evaluated and rated for fidelity 5.2 ASI group received all scheduled sessions; usual care group reported weekly services received; total weekly services for both groups reported; no significant differences reported between groups for interventions other than ASI; Attrition reported for both groups; 5.3 10% randomly selected ASI sessions subjected to fidelity checks throughout study, allowing changes to be made to insure fidelity
6. Internal validity	Independent variable is under control of experimenter. Study describes services provided in control and comparison conditions and phases. Research design provides sufficient evidence that independent variable causes change in dependent variable or variables. Participants stayed with study, so attrition is not a significant threat to internal validity
6.1. The researcher controls and systematically manipulates the independent variable. 6.2. The study describes control/comparison conditions, such as the curriculum, instruction, and interventions. 6.3. Control/comparison‐condition or baseline‐condition participants have no or extremely limited access to the treatment intervention. 6.4. The study clearly describes assignment to groups. 6.8. Overall attrition is low across groups (e.g., < 30% in a 1‐year study). 6.9. Differential attrition between groups is low (e.g., ≤10%) or is controlled for by adjusting for noncompleters.	No	6.1 Nonrandom group assignment; retrospective record review study; intervention not prospectively controlled 6.2 Control condition described and reported 6.3 ASI intervention not provided to control group 6.4 Nonrandom group assignment with comparisons between group on multiple factors; no significant differences between groups; ASI group assignment depended on group therapy being full; group therapy did not receive ASI; no discussion or consideration of potential factors that may have influenced outcomes, aside from potential maturation effects 6.8 Subject selection process from existing records reported; 24 subjects met inclusion criteria, 4 (<20%) removed due to some invalid item scores on pre–posttests 6.9 10% difference in n between groups, unspecified but likely due to loss of subjects due to invalid test items.	Yes	6.1 Intervention under experimenter control 6.2 Researchers and caregivers blind to group membership 6.3 ASI intervention provided and available only through camp‐based study 6.4 Random assignment to ASI or fine motor groups by statistician not associated with study using SPSS 6.8 10% attrition; report of 4 “drop outs”, group not specified; 4 other children removed by experimenters or parents 6.9 10% attrition, groups not reported but smaller n in fine motor group assume losses mostly in that group	Yes	6.1 Intervention under control of investigator 6.2 Control condition described and reported; additional services for both groups adequately recorded weekly and reported 6.3 Both groups received equal amounts of school‐based OT but only intervention group received ASI 6.4 Random assignment to treatment or usual control group by random permuted blocks within four strata based on cognitive level with controls for IQ and autism severity; no significant differences in preintervention demographic criteria; 6.8 Attrition less than 10%; 6.9 No attrition in intervention group; control group lost 1 to posttesting and 1 completed partial follow‐up
7. Outcome measures/dependent variables	Outcome measures are applied appropriately to gauge effect of the practice on study outcomes. Outcome measures demonstrate adequate psychometrics.
7.1. Outcomes are socially important in that, they constitute or are theoretically or empirically linked to improved quality of life, an important developmental/ learning outcome, or both. 7.2. Study clearly defines and describes measurement of dependent variables 7.3. Study reports effects of intervention on all measures of outcome targeted by the review (p levels and effect sizes [ES]) or data from which ESs can be calculated for group comparison studies. 7.4. Frequency and timing of outcome measures are appropriate. 7.5. Study provides evidence of adequate internal reliability, inter‐observer reliability, test–retest reliability, or parallel form reliability, as relevant. 7.6. Study provides adequate evidence of validity, such as content, construct, criterion (concurrent or predictive), or social validity.	No	7.1 Outcomes related to sensory, motor, cognitive measures are socially appropriate 7.2 No psychometrics presented; components of interest on measure described 7.3 Sufficient data presented to calculate effect sizes 7.4 Frequency and timing of outcome measures appropriate 7.5 No psychometric information on outcome measures reported 7.6 No psychometric information reported	Yes	7.1 Socially important GAS goals, SRS, and other outcomes. 7.2 Measures clearly described; psychometrics presented for outcome measures 7.3 All outcomes reported or data available to calculate effect sizes 7.4 Frequency and timing of outcome measures appropriate 7.5 Reliability for outcome measures reported 7.6 Validity presented when available	Yes	7.1 Goals socially appropriate, GAS goals individualized to child quality assurance of GAS goals established; other functional outcomes based on literature 7.2 Measures clearly described and strength of psychometrics detailed for all outcome measures 7.3 All measures included in results; p levels and effect sizes presented 7.4 Frequency and timing of outcome measures appropriate 7.5 Reliability for outcome measures reported 7.6 Evidence of validity of outcome measures reported
8. Data analysis	Data analysis is conducted appropriately. Study reports information on effect size (ES)
8.1 Data analysis techniques are appropriate for comparing change in performance of two or more groups. 8.3 Study reports one or more appropriate effect‐size statistics for all outcomes relevant to review being conducted, even if the outcome is not statistically significant, or provides data from which appropriate ESs can be calculated.	Partial	8.1 Analysis techniques appropriate for comparing change; large number of analyses for small sample 8.3 Sufficient data presented to allow calculation of effect sizes. Partial η2 calculated for JMAP. Effect size average below 0.25.	Partial	8.1 Use of statistical consultant; analysis appears appropriate for data 8.3 Partial η2 calculated effect sizes on all measures; effect sizes on GAS and autistic mannerisms on SRS reported; sufficient data reported to calculate effect sizes on other measures. Effect size average below 0.25	Yes	8.1 Analysis techniques appropriate for date; secondary outcomes nonnormally distributed and some nonsignificant differences between baseline scores of groups might be considered clinically relevant justified use of change scores rather than standard scores 8.3 Effect size presented for GAS, PEDI change scores, and PDDBI change scores. Effect size average above 0.25

Note. Items 6.5, 6.6, 6.7, and 8.2 removed from the table as they applied only to single‐subject studies.

UC = Usual Care; SES = socio‐economic status; SI = sensory integration; ASI = Ayres Sensory Integration; GAS = Goal Attainment Scaling; PEDI = Pediatric Evaluation of Disability Inventory; PDDBI = Pervasive Developmental Disorders Behavioral Inventory; SPM = Sensory Processing Measure; SRS = Social Responsivity Scale; SPSS = Statistical Package for the Social Sciences; JMAP = Japanese Miller Assessment for Preschoolers.