Table 3.
Quality assessment of impact evaluation
| Item | Point in time (where applicable) | Rating | |
|---|---|---|---|
| 1 | Study design (potential confounders taken into account) | End of intervention | High confidence: Randomised controlled trial (RCT), regression discontinuity design, interrupted time series, instrumental variable |
| Medium confidence: Difference in differences with matching, propensity score matching | |||
| Low confidence: Other matching | |||
| 2 | Adequate sample size | High confidence: Sample size ≥100 or cluster ≥60 | |
| Medium confidence: Sample size <100 or cluster <60 | |||
| Low confidence no power calculation or sample size <30 or cluster <30 | |||
| 3 | Losses to follow up are presented and acceptable a | End of intervention | High: attrition within Institute of Education Science (IES) bounds a |
| Medium: attrition close to IES bounds | |||
| Low: attrition not reported or attrition outside IES bounds | |||
| N/A for ex post studies | |||
| 4 | Intervention if clearly defined | High confidence: intervention clearly and fully described | |
| Medium confidence: brief description of intervention | |||
| Low confidence: intervention named but not described, or not named | |||
| 5 | Outcome measures are clearly defined and reliable | High confidence: outcome measure clearly and fully described, preferably with reference to validation | |
| Medium confidence: brief description of outcome | |||
| Low confidence: outcome named but not described | |||
| 6 | Baseline balance (not applicable for before versus after) | High confidence: RCT or baseline balance report and satisfactory (imbalance on 5 or less than 5 percent) | |
| Medium confidence: Imbalance between 5‐10 percent | |||
| Low confidence: Baseline balance not reported, or reported and lack of balance on 10 or more than 10 percent | |||
| Overall confidence in study findings | End of intervention | Lowest rating across items 1a, 4a, 6 and 7 | |