Table 6.
Specific solutions to issues encountered in preparing data for meta‐analysis.
| Study | Problem | Solution and reasoning |
|---|---|---|
| Garcia et al. (2020) and Gehring et al. (2018) | Missing sample size split between treatment and control. | Assume 50–50 split as study design is an RDD. |
| Lynch and Kim (2017) | Missing sample size split between treatment and control. | Assume even split between treatment arms as study design is an RCT. |
| Herrera et al. (2013) | Missing sample size split between treatment and control. | Take from Garcia et al. (2020) which evaluates same intervention and same sample, assume one less in total sample size is from treatment group. |
| Theodos et al. (2017) | Missing standard deviation for all non‐ITT results. | Assume same as the one derived from the respective ITT results. |
| Martin et al. (2013b) | Missing standard deviation and insufficient information to follow fully other potential solutions. | Apply method from Walter and Yao (2007) assuming that the range of the observed results (which is not reported) covers the range of potential results on the scale. |
| Modestino (2019b) | Missing prevalence of outcome among treatment and/or control group for passing tests outcome. | Assume 50% of control group passes MCAS English language/mathematics test, based on Schliemann et al. (2022). |
| Schwartz et al. (2021) | Missing prevalence of outcome among treatment and/or control group for passing tests outcome. | Use pass rate for treatment and control groups combined, then apply treatment effect to this to estimate adjusted treatment group outcome. |
| Martin et al. (2013b) | Missing mean of outcome among treatment and/or control group for education skills, confidence and self‐efficacy outcome. | Assume control group mean equal to constant from regression. |
| Kessler et al. (2022) | Not clear whether bracketed figures reported below control means in result tables are standard deviations or standard errors. | Assume they are standard deviations given magnitude in relation to mean figures. |
| Herrera et al. (2013) | Report dichotomous measures of education engagement/participation/enjoyment whilst all others report continuous measures. | Construct log odds ratio and then convert to SMD using . |
| Modestino and Paulsen (2019a) | Report dichotomous measures of socio‐emotional engagement/skills whilst all others report continuous measures. | Construct log odds ratio and then convert to SMD using . |
| Heller (2014), Davis and Heller (2020) and Heller (2022) | Report number of days attended school whilst other studies evaluating outcome report attendance rate. | Assume maximum number of school days is 178, based on Illinois State Board of Education (2022). |
| Davis and Heller (2020) | 2013 sample ineligible as for them intervention does not take place during transitional summer. | Use results from 2012 sample only. |
| Robles (2018) | Report results from multiple specifications. | Use results from 3‐nearest‐neighbours specification as nearest neighbour matching models have generally lower bias than inverse propensity score weighting, and 1‐nearest‐neighbour matching is done without replacement reducing common support. |
| Heller (2014) | Report results from multiple specifications. | Use results from the means and 0 s imputed model to minimise any potential bias introduced by missing data. |
| Heller (2022) | Report results from multiple specifications. | Use results from the means model to minimise any potential bias introduced by missing data. |
| Mariano and Martorell (2013) | Report results from multiple specifications. | Use results from English language arts/mathematics only groups as these produce the most logical comparison at the discontinuity; use results from fixed effect model as it best fits the data. |
| Johnson (2020) | Report results from multiple specifications. | Use results from DDD which reduces bias associated with unobserved characteristics. |
| Wachen (2016) | Report results from multiple specification. | Use results from linear regression model as results from logistic regression model result in highly asymmetric confidence interval when transformed to log odds ratio. |
| Henson (2018) | Report result for impact of allocation on progression to HE is logistic regression coefficient which cannot be transformed to impact of participation. | Calculate proportion of treatment and control individuals progressing to HE based on odds ratio (exponentiated coefficient), using this not the original odds ratio to calculate effect size (difference in figure is marginal whilst increasing relatability across impact types), take treatment effect as difference in proportions and transform this to estimate impact of participation, from which construct effect size. |
| Cohodes et al. (2022) and Robles (2018) | Cohodes et al. (2022) evaluates three forms (six‐week, one‐week, online) of the intervention, Robles (2018) evaluates one of these (six‐week). | When both studies are included in any analyses, the results from Robles (2018) are combined with the results from Cohodes et al. (2022) for the treatment arm that they both evaluate, before then combining the results across the different treatment arms evaluated by Cohodes et al. (2022). |
| Cohodes et al. (2022) and Robles (2018) | Evaluations of the same intervention have differing study design quality. | Use Cohodes et al. (2022) rating (high) as this is the larger of the studies and produces more result estimates. |
| Lynch and Kim (2017) | Evaluates two arms of the same intervention – one that is ‘in whole’ a summer programme and one that is the summer programme plus the provision of a laptop. | Exclude results for the summer programme plus laptop arm, as these are least representative of the true impact of the summer programme. |
Source: IES (2024).