Using a Multi-Site RCT to Predict Impacts for a Single Site: Do Better Data and Methods Yield More Accurate Predictions?

Abstract

Multi-site randomized controlled trials (RCTs) provide unbiased estimates of the average impact in the study sample. However, their ability to accurately predict the impact for individual sites outside the study sample, to inform local policy decisions, is largely unknown. To extend prior research on this question, we analyzed six multi-site RCTs and tested modern prediction methods—lasso regression and Bayesian Additive Regression Trees (BART)—using a wide range of moderator variables. The main study findings are that: (1) all of the methods yielded accurate impact predictions when the variation in impacts across sites was close to zero (as expected); (2) none of the methods yielded accurate impact predictions when the variation in impacts across sites was substantial; and (3) BART typically produced “less inaccurate” predictions than lasso regression or than the Sample Average Treatment Effect. These results raise concerns that when the impact of an intervention varies considerably across sites, statistical modelling using the data commonly collected by multi-site RCTs will be insufficient to explain the variation in impacts across sites and accurately predict impacts for individual sites.

INTRODUCTION

Motivation

The goal of evidence-based policy is to use credible evidence to inform policy decisions. A key policy decision faced by policymakers in education and other areas is whether to adopt an intervention – e.g., a new curriculum or an after-school program. Policymakers aiming to improve outcomes for a certain group of individuals should consider whether adopting various interventions would improve outcomes in their context—and improve them by enough to warrant the costs. However, the impacts of policy decisions are never known in advance. At best, policymakers have evidence that can be used to predict the impact of their policy decisions. A key question, then, is how accurately policymakers can predict the impact of their decisions – in their local context -- in advance and based on the evidence available.

To produce evidence of program impact, social policy research has often turned to multi-site, randomized controlled trials (RCTs) because they provide internally valid estimates of the effects of interventions. However, while RCTs are designed to produce unbiased estimates of average impact across all participating sites—and whatever population those sites represent—they are not designed to produce unbiased predictions of impact for individual sites. This is an important limitation when policy decisions are made locally. For example, suppose a school principal is trying to decide whether to adopt an educational program. If the impact of the intervention varies across schools, evidence on the average impact in a multi-site trial may not accurately predict the impact of adopting the intervention in the principal’s school.

Predicting the impact of an educational intervention for a single school district or school using evidence from a multi-site RCT is particularly challenging when the impact of the intervention varies across sites in the population. And such variation is not uncommon. Weiss et al. (2016) provides evidence of cross-site impact variation from 16 RCTs of 16 different interventions, finding large variation for some interventions and little or no variation for others. These results suggest that for at least some interventions, variation in impacts could challenge efforts to predict the impacts in individual sites using evidence from multi-site RCTs.

In principle, researchers can address these challenges by modelling the variation in impacts across sites and using those models to predict the impact of the intervention in any site. However, in practice, it is often challenging to build a model that makes accurate predictions. Importantly, the correct model is never known: Researchers are probably unaware of some key variables that moderate the impact of the intervention—or may be aware but the variables not available in the data—and the relationship between these moderators and impacts is never known. Facing imperfect knowledge and substantial data collection costs, researchers conducting RCTs may fail to collect and include some important site-level impact moderators from their models. In addition, building a model that makes accurate out-of-sample predictions is challenging because the relatively small number of sites included in most multi-site RCTs restricts the number of site-level moderator variables that can be included in any one model. Moreover, multi-site RCTs are generally powered to only detect an overall average effect and may have limited ability to estimate moderation effects with precision. Selecting the most important moderators to include in the model may also be challenging without strong prior evidence, which researchers conducting multi-site RCTs often lack. Finally, suppose that all these challenges can be overcome, and researchers are able to build a rich model that produces accurate predictions of impact for individual sites, conditional on site-level moderators collected as part of the study. This model would still be unusable for out-of-sample prediction unless data on the same moderators are available for sites outside of the study sample. In summary, there are serious challenges to building models that are useful for predicting the impact of interventions in individual sites outside the RCT sample—and limited empirical evidence on how well this can be done in practice.

In recent work, Orr et al. (2019) found that the estimated average impact in the sample from an RCT generally produced inaccurate impact predictions for individual sites, and that basic models developed to account for variation in impacts across sites did not improve the accuracy of those predictions. Specifically, they developed and tested two-level hierarchical linear models of students nested within sites, where the impact for an individual site was specified to be a linear function of site-level moderators—in particular, moderators that were typically publicly available to ensure that they would be known to local policymakers and thus usable for predicting local impacts. These models were used to predict site-level impacts in three RCTs of interventions targeted at children and youth: (1) educational technology, (2) charter schools, and (3) Head Start. The paper demonstrated that using the estimated average impact in the sample as the predicted impact for individual sites generated large prediction errors: The root mean squared prediction errors ranged from 0.07 to 0.35 standard deviations (measured in the same effect size units as the study-reported impacts). These differences likely reflect variation in impacts across sites, with large prediction errors when the cross-site impact variance is high and small prediction errors when the cross-site impact variance is low. In addition, the paper found that models with one, two, or five moderators did not yield impact predictions for individual sites that were more accurate than simply using the estimated average impact.

However, a few limitations of Orr et al. (2019) suggest that additional research is warranted. First, the study examined only three RCTs; evidence from additional studies is needed to build a broader evidence base on this topic. Second, the simple linear models that it used did not take advantage of recent advances in predictive models, especially for contexts with limited sample sizes and large numbers of predictors. Third, Orr et al. (2019) focused exclusively on publicly available characteristics of schools when building models, based on the argument that these variables would be known to local policymakers when considering whether to implement the intervention, and excluded other types of moderators that may explain impact variation (Weiss, Bloom, & Brock, 2014). However, in some circumstances, local policymakers may be able to predict these other moderators in advance, such as the characteristics of the students likely to participate in the intervention and features of the counterfactual condition faced by students locally. If so, excluding these variables could have led to overly pessimistic conclusions about the potential for using multi-site RCTs to make accurate out-of-sample impact predictions for individual sites.

Contribution of the Paper

This study extends prior research in three ways. First, it expands the scope and likely generalizability of the evidence base by analyzing a larger number of multi-site RCTs. In addition to reanalyzing data from the evaluation of charter schools examined in Orr et al. (2019) (Gleason et al. 2010), it analyzes data from evaluations of Teach for America (Clark et al., 2013), the Teaching Fellows Program (Clark et al., 2013), the U.S. Department of Education’s Student Mentoring Program (Bernstein et al., 2009), the Gang Resistance Education and Training Program (G.R.E.A.T., Esbenson et al., 2012), and a summer reading program (Wilkins et al., 2012).

Second, the paper improves on the analysis in Orr et al. (2019) by testing two modern statistical methods for making out-of-sample predictions: (1) lasso regression (Tibshirani, 1996) and (2) Bayesian Additive Regression Trees (BART; Chipman et al., 2010). Orr et al. (2019) used standard linear regression models with moderator variables selected based on p values. However, this approach risks producing models that are overfit to the sample data and yield poor out-of-sample predictions. In contrast, lasso and BART are designed for making out-of-sample predictions and thus may be better suited to informing local policy decisions in sites not included in the evaluation. Lasso’s use of cross-validation to select variables and shrink some coefficients toward zero suggests that it may help to avoid overfitting that can lead to inaccurate out-of-sample predictions.¹ In addition, lasso has been shown to work well in relatively small sample sizes (as few as 20), with more possible parameters (p) than observations (n) (e.g., Eberlin et al. 2014, Finch and Finch 2016, Donaho and Stodden 2006). BART’s main strength is its ability to uncover patterns in the data, including nonlinear relationships and high-order interactions, without having to assume a specific functional form for the model or to pre-specify which interactions to include. Kern et al. (2016) found that BART performed better than most other methods in a variety of scenarios when generalizing impact estimates to target populations.

Third, this study uses a more comprehensive range of site-level moderators to predict the impact in individual sites. In particular, it examines the extent to which data from multi-site trials on four different types of effect moderators—individually or together—can improve the ability of statistical models to predict the impact of interventions in individual sites. In addition to publicly available information on schools and districts used by Orr et al. (2019) (e.g., size, student composition, resources, and location), this study uses data collected by the original study teams on three different types of moderators: (1) characteristics of students participating in the study, (2) features of the intervention or how it was implemented, and (3) features of the counterfactual condition faced by individuals assigned to the control group. We assess whether models that include all four types of moderators yield more accurate predictions than models that are restricted to just publicly available information on schools and districts.

Fourth, this paper takes a new approach to estimating the magnitude of the likely prediction errors. This approach, detailed below, uses random effects meta-analysis methods to estimate the (root) mean squared prediction error as a function of the bias, sampling error in the predicted impacts, and cross-site heterogeneity in impacts.

We focus on the magnitude of prediction errors because local policymakers are in a better position to make policy decisions that will improve student outcomes if informed by impact predictions with small errors than by impact predictions with large errors. While Orr et al. (2019) translates prediction errors into estimates of their consequences for decision-making—specifically, on whether policymakers will make the “correct” decision about whether to adopt an intervention based on the evidence—we focus here exclusively on the magnitude of the prediction errors and save the policy consequences of these errors for future research.

THEORY

The goal of this study is to test different methods for predicting the true impact of an intervention in an individual site, and to assess the expected accuracy of those predictions from a randomly selected site from the population. Let ${\delta }_j$ be the estimand of interest—the true impact in site $j$, for $j=1,\dots ,N$. For a particular prediction method, let ${\delta }_j^p$ be expected value of the site-specific impact prediction—that is, the parameter to which the predicted impact for site $j$ converges as the number of sites in the multi-site RCT used to estimate the prediction model converges to infinity. Under these assumptions, the bias of the prediction method for site $j$ equals $b_j\equiv {\delta }_j^p-{\delta }_j$. We would in general expect the prediction models to be biased unless the variance of the true impact across sites--${\tau }^2$—equal zero or the assumptions of the model explaining variation in impacts strictly hold.

However, the accuracy of the predicted impact for an individual site depends on both the bias and the variance of those prediction. Since the predictions come from a multi-site RCT, the estimated parameters of the prediction model will contain sampling error that depends on the number of sites included in the RCT. Let ${\widehat{\delta }}_j^p$ be the predicted impact from a finite sample and ${\epsilon }_j^p$ be the random error in the model’s prediction for site $j$. We model this predicted impact with the equation below:

$${\widehat{\delta }}_j^p={{\delta }_j^p+{\epsilon }_j^p=\delta }_j+b_j+{\epsilon }_j^p,$$ (1)

where ${\delta }_j~\left(\delta ,{\tau }^2\right),b_j~\left(b,{\sigma }_b^2\right)$, and ${\epsilon }_j^p~\left(0,{\sigma }_p^2\right)$.

These assumptions allow us to define a measure of the accuracy of the prediction method applied to a multi-site RCT with a finite number of sites for a randomly selected site in the population. The root mean squared prediction error ($RMSPE$) can be expressed as follows:

$$RMSPE=\sqrt{b^2+{\sigma }_b^2+{\sigma }_p^2}$$ (2)

In words, the root mean squared error ($RMSPE$) is the square root of the sum of the squared average bias of the prediction method ($b^2$), the variance of the bias across sites (${\sigma }_b^2$), and the error variance of the prediction error from estimating the prediction model in a finite sample (${\sigma }_p^2$). This formulation will allow us to estimate the RMSPE, as described later, by separately estimating the average bias in the predictions ($b$) and the sum of the variances of the remaining two components of the prediction error (${\sigma }_b^2+{\sigma }_p^2$).

DATA

We used data from six large-scale multi-site RCTs of interventions focused on improving youth outcomes (Table 1). These six RCTs can be thought of a convenience sample because they were chosen largely based on data availability. These RCTs fall within a larger set of studies that are highly visible to policymakers through government reports and evidence clearinghouses. Each of these studies offers an opportunity to test how well we can use the RCT data to make accurate impact predictions for individual sites.

Table 1 –

RCTs Included in the Analysis

Intervention	Outcome Domains	Study Design		Data Source
		Unit of Assignment	Number of Sites
Gang Resistance Education and Training Program (G.R.E.A.T.)	Behavioral	Classroom (N=195)	31 schools	ICPSR
Teach for America	Achievement	Student (N≈4,570)	45 schools	NCES
The Teaching Fellows Program	Achievement	Student (N≈4,120)	44 schools	NCES
The Student Mentoring Program	Behavioral, achievement, performance in school	Student (N≈1,970)	32 program grantees	NCES
Charter Middle Schools	Achievement	Student (N≈2,150)	36 schools	NCES
Summer Reading	Achievement	Student (N≈1,520)	112 schools	NCES

Note: Student sample sizes are rounded to the nearest 10 for compliance with non-disclosure procedures for NCES restricted use data.

Student- and site-level data from these studies were used to build site-level analysis files that contain site-level impact estimates for student outcomes and site-level moderator variables that may predict the size and direction of the site-level impacts. Consolidating the student-level data into site-level datasets simplified the application of lasso regression and BART for making site-specific impact predictions without losing any information helpful in predicting the impact in individual sites.²

Each of the six RCTs included multiple sites in the sample and conducted random assignment within sites, allowing us to produce unbiased impact estimates for each site. From a methodological perspective, we defined a “site” as the randomization block in which students (or in one case, classrooms) were randomly assigned. For all but two of the studies, the site was defined as the public school in which the intervention was implemented. For the evaluation of charter middle schools, we defined the site as the collection of public schools from which students applied to a single charter school--or a pair of nearby charter schools with multiple students applying to both—and were either admitted to a charter school or not admitted based on a random lottery. For the evaluation of the Student Mentoring Program, we defined the site as the grantee organization—often a school or school district—that supported the program locally, and within which students were randomly assigned.³

For each RCT, we used data on one or more student outcomes for which the original study authors estimated and reported the impacts of the intervention. For measures of student achievement based on standardized tests, the student-level outcome variable ($y_{ij}$) was standardized by the original study authors to have a mean of zero and standard deviation of 1 in a defined norming population.⁴ For other continuous or ordinal outcome variables (e.g., the count or number of incidents of a particular type), we created a binary outcome variable equal to 1 or “yes” (e.g., the student had more than one incident) or 0 or “no” (e.g., the student had zero incidents) if the vast majority of students in the sample had no such incidents; otherwise, we standardized the outcome by subtracting off the mean and dividing by the standard deviation for all treatment and control students included in the analysis. Binary outcomes from the original study data (e.g., scored above the state threshold for proficiency in math or reading) were left in their natural scale (0 for no or 1 for yes). See Table 2 for a description of the student outcome variables from the original studies that we used in the analysis along with summary statistics and sample sizes.⁵

Table 2–

Student Outcome Variables Included in the Analysis

Outcome Variable	Description and Scale	Mean	SD	N
Charter Middle Schools
Math achievement, year 1	Standardized score on state test	0.38	1.01	2,130
Math achievement, year 2	Standardized score on state test	0.44	1.14	2,040
Reading achievement, year 1	Standardized score on state test	0.43	0.96	2,150
Reading achievement, year 2	Standardized score on state test	0.39	0.96	2,040
Gang Resistance Education and Training Program (G.R.E.A.T.)
Delinquency (frequency)	Number of incidents of delinquency	0.00	1.00	3,564
Delinquency (variety)	Number of different types of delinquency	0.00	1.00	3,564
Violence (any)	=1 if any violent incidents, =0 otherwise	0.12	0.32	3,564
Student Mentoring Program
Absentee rate	Ratio of absences to days enrolled	0.00	1.00	2,160
Truancy rate	Ratio of unexcused absences to days enrolled	0.00	1.00	1,440
Math grades	=1 (F), 2 (D), 3, (C), 4 (B), or 5 (A)	0.00	1.00	2,000
English grades	=1 (F), 2 (D), 3, (C), 4 (B), or 5 (A)	0.00	1.00	2,010
Science grades	=1 (F), 2 (D), 3, (C), 4 (B), or 5 (A)	0.00	1.00	1,950
Social studies grades	=1 (F), 2 (D), 3, (C), 4 (B), or 5 (A)	0.00	1.00	1,890
Math proficiency	=1 if proficient based on state test, = 0 otherwise	0.46	0.5	1,990
Reading proficiency	=1 if proficient based on state test, = 0 otherwise	0.49	0.5	2,030
Misconduct (any)	=1 if at least one incident of misconduct, =0 otherwise	0.26	0.44	1,730
Misconduct (repeated)	=1 if multiple incidents of misconduct, =0 otherwise	0.16	0.37	1,730
Delinquency (any)	=1 if at least one incident of delinquency, =0 otherwise	0.20	0.40	1,780
Delinquency (repeated)	=1 if multiple incidents of delinquency, =0 otherwise	0.11	0.31	1,570
Summer Reading Program
Reading achievement	Standardized score on study-administered test (Scholastic Reading Inventory)	0.00	1.00	1,570
Teach for America
Math achievement	Standardized score on state test or study-administered test (NWEA)	−0.58	0.93	4,570
Teaching Fellows Program
Math achievement	Standardized score on state test or study-administered test (NWEA)	−0.38	1.07	4,120

Notes: Standardized scores on state tests were standardized by the original study authors as z-scores, subtracting off the mean and dividing by the standard deviation for all tested students in the state in the same subject and grade level. Standardized scores on study-administered tests were standardized (1) relative to a national norming population for Teach for America and the Teaching Fellows Program and (2) relative to the pooled sample of treatment and control students for the summer reading program. Student sample sizes for all studies except G.R.E.A.T. are rounded to the nearest 10 for compliance with NCES non-disclosure procedures.

Finally, we used data on four types of potential impact moderators that were available in the restricted use files for the six multi-site RCTs. Some of these variables come from public sources; others were collected by the original study teams:

1. Publicly available information on the characteristics of each site. The Common Core of Data (CCD) provides publicly available information on all public schools nationwide. For the four studies that assembled CCD data on participating schools, we used data on size, student demographic composition, location, and resources to capture the context in which educational interventions are implemented.

2. Information collected on the subset of students who participated in the study. Most impact evaluations in education collect data on the characteristics of the students who participated in the study, especially their demographic characteristics, to conduct subgroup analyses. For all six studies, we created site-level averages of the participant characteristics that were collected (e.g., disability status, race/ethnicity, English learner status, and eligibility for free or reduced-price meals) to capture the types of students who would receive the intervention in each site if the intervention were implemented locally.

3. Features of the intervention or how it was implemented. Many RCTs collect measures related to the intervention itself. For the four studies that collected such measures, we used study-collected measures of intervention features or how the intervention was implemented. These included characteristics of the charter schools attended by students who were admitted by lottery, the characteristics of Teach for America and Teaching Fellows Program teachers for students randomly assigned to their classes, and the characteristics of the local mentoring programs to which students were randomly assigned.

4. Features of the “counterfactual condition.” Since impacts result from the contrast between the intervention condition and the counterfactual condition, some RCTs also collect site-level measures of the counterfactual condition experienced by students in the control group. We used study-collected measures of the counterfactual from the four studies that constructed such measures. These include measures of academic programming in the regular public schools attended by students who “lost the lottery” to attend a charter school, characteristics of teachers who taught in the same schools and grade levels as teachers from Teach for America or the Teaching Fellows Program, and participation in mentoring programs by students who were assigned the control group in the evaluation of the U.S. Department of Education’s Student Mentoring Program.

The number of moderator variables included in the analysis for each RCT is provided in Table 3. The definition of each of these moderators is provided in the Online Appendix.

Table 3 –

Number of Moderator Variables Included for Each Intervention

Intervention	Site Characteristics	Student Characteristics (Site-Level Averages)	Intervention Features	Counterfactual Features
Charter Middle Schools	21	21	90	90
G.R.E.A.T.	0	67	0	0
Student Mentoring Program	0	261	422	5
Summer Reading Program	0	14	0	0
Teach for America	23	45	202	190
Teaching Fellows Program	23	45	202	190

Note: Zero variables were included of a particular type when none were available in the original study data.

METHODS

This study used three types of methods to assess the potential for multi-site RCTs to accurately predict impacts for individual sites outside the study sample: (1) methods for estimating the true impact in a single site—to serve as a benchmark for assessing different prediction methods, (2) methods for predicting the impact in that site using only data from other sites, and (3) methods for assessing the accuracy of those predictions. All three types of methods are described below.

Methods for Estimating the True Impact in a Single Site

For each of the six multi-site RCTs, we took advantage of their experimental design to estimate the following model and produce an unbiased estimate of the impact of the intervention in each site j on student outcome $y_{ij}$:

$$y_{ij}=\alpha +\beta x_{ij}+{{\delta }_j^{w}T}_{ij}+e_{ij},e_{ij}~N\left(0,{\sigma }_j^2\right)$$ (3)

where $y_{ij}$ is the outcome for student i in site j; $x_i$ is a pre-intervention measure of the outcome for student i in site j; $T_{ij}$ is the treatment indicator that equals 1 if student i in site j was assigned to the treatment group and 0 if this student was assigned to the control group; and $e_{ij}$ is a random error term that varies across the students in site j.⁶ Using the full study sample allowed the original study authors to include a large set of covariates in the models. However, we estimated separate impact regression models for each site, based on a small fraction of the total sample, necessitating a more parsimonious model. In estimating impacts separately by site, we included a single covariate—a pre-intervention value of the outcome—given prior evidence that pre-test measures are more predictive of student outcomes than other potential covariates (Bloom, Richburg-Hayes, & Black, 2007).

The parameter ${\delta }_j^w$ reflects the intervention’s impact in site j, where the “w” superscript denotes that the impact is the impact on students within site j. We estimated this parameter using (1) ordinary least squares if the original study authors did not construct and use weights in the impact analysis or (2) weighted least squares when the study authors did construct and use weights in the impact analysis (typically to address nonresponse bias, variation in the probability of assignment to the treatment, or both).⁷

Methods for Predicting the Impact for a Single Site

In broad terms, we tested three methods for predicting the impact of an educational intervention for a site outside of the study sample but that is similar to those in the study sample: (1) the simple average of the site-level impact estimates for sites in the study sample; (2) lasso regression; and (3) BART. The first approach does not use data on site-level moderators; the second and third approaches use site-level moderator data to produce customized impact predictions for individual sites. With all three methods, we excluded a single site, estimated the prediction models in the remaining J-1 sites, and used the estimated prediction model—with the site moderator values in the excluded site, for lasso and BART—to predict the impact in the excluded site. The analysis plan was initially published on the Open Science Framework before any analysis was conducted (Olsen et al., 2021).⁸

For the simple average approach, we took the unweighted average of the unbiased site-level impact estimates and used this average as the predicted impact in the excluded site (referred to below as the “focal” site).⁹ This approach is motivated by evidence-based policymaking that identifies interventions that “work” and promotes them for local adoption.

We used lasso regression to fit a linear regression model of the intervention’s impact as a function of site-level moderators, and we used the model to predict the impact of the intervention in the excluded site. The lasso was used to penalize the model for large regression coefficients that might improve the model fit in the sample at the expense of lower out-of-sample prediction accuracy. Specifically, we used cross-validation to set the penalty term (also called the “tuning parameter”) to minimize the mean squared prediction error. The resulting linear model was used to predict the impact of the intervention in the excluded site, conditional on the site-level moderators in the model.

Our third prediction approach, BART, fits a series of regression trees of the intervention’s impact as a function of site-level moderators and then uses the set of trees to predict the impact of the intervention in the excluded site. Regression trees sequentially divide the sample into categories, based on binary and continuous variables, to explain variation in the variable being predicted—in this case, the impact of the intervention in individual sites. These trees end in multiple “leaves,” or categories defined by multiple variables (e.g., continuous variable X1 > C and binary variable X2 = 0). In our application, the predicted impact for an individual site was set equal to the average impact estimate among the collection of sites in the same category (i.e., at the same leaf). We chose to test a regression tree-based approach to prediction because, unlike standard linear regression models, they are designed for scenarios where the variables used for classification (here, impact moderators) may have non-linear or interacting relationships with the outcome (here, the estimated impact of an education intervention).

While many approaches to regression trees exist, we used BART because it is well designed to address the overfitting problem of some random forest approaches that can lead to poor predictions. To avoid producing overly complex trees that overfit the sample data, BART constructs a series of small trees and combines them into an “ensemble,” fit using a Bayesian framework with default prior distributions on key parameters (Witten, Hastie, & Tibshirani, 2013). For more details on lasso and BART, and our approach to implementing them, see the Appendix.

For both lasso and BART, we implemented a pre-modeling variable selection step, restricting the models to moderators with a correlation of at least 0.20 with the estimated impact. This step was taken to focus the algorithms on variables likely to moderate the impact of the intervention. Our original plan was to offer lasso and BART the full set of potential moderators, regardless of their correlation with the impact and without any prior judgment by us about their relative importance—leaving it entirely to the lasso and BART algorithms to choose the variables that contribute most to producing accurate impact predictions in individual sites. However, we discovered, in testing our approach to the lasso in charter school data without the 0.20 correlation screen, the lasso did not select any predictor variables for the model.¹⁰ In contrast, setting a minimum correlation of 0.20 led the lasso to select some predictor variables and allowed us to test whether models with predictors selected by the lasso method yielded more accurate predictions than ‘models’ with no predictor variables, as reflected in the simple average of the site-level impact estimates.

Methods for Assessing the Accuracy of the Predicted Impact for a Single Site

To assess the accuracy with which each of the three methods described above predict the impact of each intervention in a single site, we used and adapted the analytic approach developed in Orr et al. (2019):

• Step 1: Designate one site from the RCT as the focal (or “excluded”) site. We pretend that this site—site j—did not participate in the evaluation and is the site for which a policy decision is being made, and thus for which an impact prediction is needed.

• Step 2: Produce an unbiased estimate of the impact in the focal site. Using only the data from the focal site and methods described earlier, we estimated equation (1). This produced an unbiased estimate of the true impact in site j.

• Step 3: Predict the impact in the focal site using data from the other J-1 sites (${\widehat{\delta }}_j^p$). We fit each of the three prediction methods described earlier using the data from the other J-1 sites and then used that model, and moderator values from site j, to predict the impact in site j.

• Step 4: Estimate the prediction error. The true prediction error for site j equals ${{PE}_j=\widehat{\delta }}_j^p-{\delta }_j^w$, where ${\delta }_j^w$ is the true impact in that site. We estimated the prediction error for site j, ${\widehat{PE}}_j$, by taking the difference between the predicted impact, ${\widehat{\delta }}_j^p,$ and the unbiased estimate of the impact in site j, ${\widehat{\delta }}_j^w:{{\widehat{PE}}_j=\widehat{\delta }}_j^p-{\widehat{\delta }}_j^w$.

• Step 5: Repeat steps 1-3 for each of the remaining sites. This treats each site in the sample as the focal site for a local policy decision and estimates the prediction error for that site.

• Step 6: Estimate the Root Mean Squared Prediction Error (RMSPE), adjusted for sampling error in the focal site, across all sites in the RCT. We calculated the RMSPE instead of the mean prediction error to ensure that positive errors do not offset negative errors. To calculate the RMSPE, we conducted a random effects meta-analysis. This meta-analysis accounts for sampling error in the unbiased impact estimate for each site.

Figure 1 provides an illustration of the site-level estimates used to estimate the RMSPE for different combinations of study, outcome variable, prediction method, and sets of moderator variables. This figure displays for the sites in the Teach for America study the site-level unbiased impact estimates and the predicted impacts generated using BART with all four types of moderator variables.

Figure 1.

Comparison of Predicted Impact to Unbiased Impact Estimate for Individual Sites in the Teach for America Study: BART Predictions with All Four Types of Moderators

The meta-analysis builds on the model developed earlier in the Theory section of the paper. To conduct the meta-analysis, we regressed the estimated prediction error (${\widehat{PE}}_j$) on an intercept term. The random effects meta-analysis estimated the cross-site variance of the prediction error, excluding the within-site variance of the unbiased impact estimate ${\widehat{\delta }}_j^w$).¹¹ To estimate the RMPSE, we took the square root of the sum of two terms: (1) the square of the estimated intercept, which can be interpreted as the average bias in the prediction method, and (2) the estimated cross-site variance from the meta-analysis, which can be interpreted as the sum of the variance of the prediction bias across sites and the variance of the predicted impacts that results from estimating prediction models in finite samples. The estimated RMSPE provides an estimate of the prediction error that can be expected for a randomly selected site from the population represented by sites that participated in the RCT.

To assess the precision of the RMSPE estimates and the differences between RMSPEs from different models, we used the bootstrap method. In particular, 1,000 bootstrap samples of J sites were selected with replacement—where J equals the number of sites included in the RCT— estimated the RMSPE for each bootstrap sample, and we concluded that prediction method A yielded a lower RMSPE than prediction method B if the estimated RMSPE was lower for A than for B in 950 or more of the 1,000 bootstrap samples. Similarly, we concluded that prediction method B yielded a lower RMSPE than prediction method A if the estimated RMSPE was lower for B than for A in 950 or more of the 1,000 bootstrap samples. In effect, we used bootstrapping to test for significant differences at the 90 percent level with a two-tailed test.

Finally, we assessed the magnitude of the RMSPE estimates by comparing them to external estimates of the average impacts of educational interventions. This comparison was made to determine whether the prediction errors were large enough to make an intervention with zero impact appear to have an average impact—one of average magnitude—or an intervention with an average impact appear to have a zero impact. For the four studies that reported impacts on achievement outcomes—the evaluations of charter schools, Teach for America, the Teaching Fellows Program, and the summer reading program—we compared the RMSPE estimates to a threshold value of 0.04 standard deviations, the average impact across 1,352 estimates from randomized trials of the effects of educational interventions on broad tests of student achievement in mathematics or English language arts (Kraft, 2020).¹² For the evaluation of G.R.E.A.T., we compared the RMSPE estimates to a threshold value of 0.09, the average effect size across 68 estimates of the effects of school-based programs for preventing problem behaviors in middle school (Wilson, Gottfredson & Najaka, 2001).¹³ For the evaluation of the student mentoring program, we compared the RMSPE estimates to a threshold value of 0.21, the average effect size across 83 estimates of the effects of mentoring programs for youth from randomized trials and non-randomized comparison group designs (DuBois et al., 2011).¹⁴ Lastly, the evaluations of G.R.E.A.T. and the student mentoring program reported impacts for some binary outcomes. Because the published evidence from meta-analyses generally focused on continuous outcomes, we lack evidence on the average effects of educational interventions on binary outcomes. For this reason, we simply report the RMSPEs on these outcomes and leave it to readers to make their own assessment of whether prediction errors implied by these estimates should be considered large or small.

FINDINGS

To provide context for the main findings, Table 4 provides estimates of the unconditional cross-site standard deviation of the impact for each study and outcome variable. These estimates indicate the challenge that prediction methods face in producing accurate impact estimates. When the unconditional cross-site standard deviation of impact is large, the only way to produce accurate impact predictions for individual sites is through a model that explains a sizable share of the impact variation across sites.

Table 4–

Cross-Site Standard Deviation of Impact

Outcome Variable	Scale at the Student Level	Cross-Site Standard Deviation of Impacts
Charter Middle Schools
Math achievement, year 1	Continuous (standard deviations)	0.161
Math achievement, year 2	Continuous (standard deviations)	0.311
Reading achievement, year 1	Continuous (standard deviations)	0.184
Reading achievement, year 2	Continuous (standard deviations)	0.192
Gang Resistance Education and Training Program (G.R.E.A.T.)
Delinquency (frequency)	Ordinal (standard deviations)	0.000
Delinquency (variety)	Ordinal (standard deviations)	0.000
Violence (any)	Binary (0 or 1)	0.045
Student Mentoring Program
Absentee rate	Continuous (0-1)	0.001
Truancy rate	Continuous (0-1)	0.001
Math grades	Ordinal (1, 2, 3, 4, or 5)	0.000
English grades	Ordinal (1, 2, 3, 4, or 5)	0.000
Science grades	Ordinal (1, 2, 3, 4, or 5)	0.038
Social studies grades	Ordinal (1, 2, 3, 4, or 5)	0.043
Math proficiency	Binary (0 or 1)	0.000
Reading proficiency	Binary (0 or 1)	0.000
Misconduct (any)	Binary (0 or 1)	0.019
Misconduct (repeated)	Binary (0 or 1)	0.008
Delinquency (any)	Binary (0 or 1)	0.000
Delinquency (repeated)	Binary (0 or 1)	0.000
Summer Reading Program
Reading achievement	Continuous (standard deviations)	0.231
Teach for America
Math achievement	Continuous (standard deviations)	0.174
Teaching Fellows Program
Math achievement	Continuous (standard deviations)	0.321

Table 4 shows that the six studies cover the range of the cross-site impact variation from prior studies. Weiss et al. (2016) report estimates of cross-site standard deviation of impacts ranging from 0 for an afterschool math program to 0.35 for math achievement in year 2 from the same charter school evaluation analyzed in this study. Similarly, the cross-site standard deviation reported in Table 4 ranged from 0 for several outcomes in the evaluations of G.R.E.A.T. and the student mentoring program to 0.31 for math achievement in year 2 for the evaluation of charter schools.

The first question we addressed is whether a study’s estimate of the average impact produces accurate predictions of impact for individual sites. Table 5 presents the RMSPE estimates for the prediction method that uses the simple average of the impact estimates in the other sites. For the four studies that focused on impacts on student achievement measures—studies of charter middle schools, a summer reading program, Teach for America, and the New Teachers Program—the RMSPE estimates are indicative of large prediction errors, ranging from 0.13 for the impact of Teach for America on math achievement to 0.33 standard deviations for the impact of charter schools on math achievement in year 2. These RMSPE estimates are all greater than 0.04, the average impact of educational interventions cited earlier (Kraft, 2020). This suggest that using the simple average of the impact estimates in other sites yields predictions with errors that are large enough to mislead local policymakers into reaching the wrong conclusion about whether the program would be effective locally.

Table 5–

RMSPE Estimates from the Simple Average of the Impact Estimates in the Other Sites

Outcome Variable	Scale at the Student Level	RMSPE
Charter Middle Schools
Math achievement, year 1	Continuous (standard deviations)	0.17
Math achievement, year 2	Continuous (standard deviations)	0.32
Reading achievement, year 1	Continuous (standard deviations)	0.19
Reading achievement, year 2	Continuous (standard deviations)	0.20
Gang Resistance Education and Training Program (G.R.E.A.T.)
Delinquency (frequency)	Ordinal (standard deviations)	0.02
Delinquency (variety)	Ordinal (standard deviations)	0.02
Violence (any)	Binary (0 or 1)	0.05
Student Mentoring Program
Absentee rate	Continuous (0-1)	0.03
Truancy rate	Continuous (0-1)	0.02
Math grades	Ordinal (1, 2, 3, 4, or 5)	0.04
English grades	Ordinal (1, 2, 3, 4, or 5)	0.02
Science grades	Ordinal (1, 2, 3, 4, or 5)	0.04
Social studies grades	Ordinal (1, 2, 3, 4, or 5)	0.04
Math proficiency	Binary (0 or 1)	0.01
Reading proficiency	Binary (0 or 1)	0.02
Truancy rate	Continuous (0-1)	0.02
Misconduct (any)	Binary (0 or 1)	0.03
Misconduct (repeated)	Binary (0 or 1)	0.01
Delinquency (any)	Binary (0 or 1)	0.02
Delinquency (repeated)	Binary (0 or 1)	0.03
Summer Reading Program
Reading achievement	Continuous (standard deviations)	0.26
Teach for America
Math achievement	Continuous (standard deviations)	0.12
Teaching Fellows Program
Math achievement	Continuous (standard deviations)	0.15

In contrast, for the other two studies, the findings suggest that the impact of the intervention in a site can be predicted accurately by taking the simple average of the impact estimates in the other sites. For the evaluation of G.R.E.A.T., the RMSPE estimates in Table 5 for outcomes measured in standard deviation units were close to zero and smaller than the average effect size of 0.09 found for other middle school-based prevention programs (Wilson, Gottfredson & Najaka, 2001). For the evaluation of a federal student mentoring program, the RMSPE estimates for outcomes measured in standard deviation units were generally close to zero and uniformly much smaller than the average effect size of 0.21 found for similar programs (DuBois et al., 2011). Together, the estimates from these two studies reflect prediction errors that are too small to mislead local policymakers into reaching the wrong conclusion about the likely impacts on these outcomes locally.¹⁵

The second question we addressed is whether the advanced statistical methods we tested, lasso and BART, combined with different sets of potential moderators, produced more accurate impact predictions than the simple average of the impact estimates in the other sites, and predictions accurate enough to support local policymaking. To address this question, we analyzed data from the four studies that estimated the impact of the intervention on measures of student achievement, and where the simple average consistently yielded inaccurate impact estimates, as evidenced by RMSPE estimates of greater than the threshold that we established earlier: (1) charter middle schools, (2) Teach for America, (3) the Teaching Fellows Program, and (4) the summer reading program. However, most analyses were restricted to the three studies that collected data on all four types of moderators: (1) charter middle schools, (2) Teach for America, and (3) the Teaching Fellows Program.¹⁶ We tested the performance of lasso and BART relative to the simple average approach and relative to each other, separately for each of the four types of moderator variables described earlier as well as for all four moderator types together.

The results (Figures 2–6) provide little evidence that lasso regression produces more accurate impact predictions for individual sites than the simple average of the impact estimates in the other sites. The method did not produce consistently smaller RMSPE estimates than the simple average of the impact estimates in the other sites for either the four types of moderator variables individually (Figures 2, 3, 4, and 5) or for all four types together (Figure 6). Lasso regression increased the RMSPE for the Teaching Fellows Program for all tested combinations of moderator variables¹⁷; for the other two studies, it increased the RMSPE for some outcomes and types of moderators and reduced it for others.¹⁸

Figure 2. RMSPE for Different Prediction Methods Using Site Characteristics as Moderators.

Note: ‘*’ identifies scenarios in which no moderator variables were offered to lasso or BART because none of them had a correlation of at least 0.20 with the outcome.

Figure 6.

RMSPE for Different Prediction Methods: All Four Types of Moderators

Figure 3. RMSPE for Different Prediction Methods Using Aggregate Study Participant (Student) Characteristics as Moderators.

Note: ‘*’ identifies scenarios in which no moderator variables were offered to lasso or BART because none of them had a correlation of at least 0.20 with the outcome.

Figure 4.

RMSPE for Different Prediction Methods Using Intervention-Related Characteristics as Moderators

Figure 5.

RMSPE for Different Prediction Methods Using Counterfactual-Related Characteristics as Moderators

The performance of BART is more mixed, generally reducing the RMSPEs for the charter middle school study but not for the other three studies. For charter middle schools, BART produced predictions with lower RMSPE than the simple average prediction method for 13 of the 15 predictions across the different outcomes and sets of moderator variables. Of these 13 differences, 12 were statistically significant. In contrast, for the other three studies—Teach for America, the Teaching Fellows Program, and the summer reading program—BART produced lower RMSPE for some predictions and higher RMSPE for others. These differences were statistically significant for only one of the nine BART predictions—roughly as we should expect if the differences were purely due to random chance instead of systematic differences in the performance of different methods.

However, even with BART, the prediction errors were large enough to potentially make effective interventions appear ineffective and vice versa. For the four studies in which the simple average prediction method produced RMSPE estimates of greater than 0.04 standard deviations—the average impact of educational interventions on broad measures of student achievement—BART was never able to reduce the RMSPE below 0.04 standard deviations. The smallest RMSPE generated by BART was 0.09 standard deviations for predictions of the impact of Teach for America on math achievement using all four types of moderators.

Figures 7 and 8 show how much the prediction accuracy of lasso and BART models improve with access to data from all four types of moderators versus data from only a single type of moderator. For lasso, Figure 7 shows that prediction accuracy does not typically improve from having access to all four types of moderators: The RMSPE from the models with all four types (as denoted with the vertical bar) is sometimes larger and sometimes smaller than the RMSPE from models with a single type. In contrast, for BART, Figure 8 shows that prediction accuracy often but not always improves from having access to all four types of moderators. For the charter middle school study, the reductions in RMSPE from BART were greater with all four types of moderators than with just a single type. These differences were statistically significant for 8 of 11 combinations of outcome variable and type of moderator variable. This suggests that for this one study, BART predictions benefit from having access to a wider range of moderator types. This pattern generally held for the study of Teach for America: The estimated RMSPE was lower with all four moderator types than for any single type of moderator, but only one of those four differences were statistically significant. For the study of the Teaching Fellows Program, the results were mixed: The estimated RMSPE was lower with all four moderator types than the RMSPEs with two of the four moderator types but higher than the RMSPEs with the other two types of moderators. Finally, this analysis could not be completed for the summer reading study because the data only contained one type of moderator (characteristics of the students who participated in the study).

Figure 7. RMSPE with All Four Types of Moderators: Lasso.

Note: ‘*’ identifies scenarios in which no moderator variables were offered to lasso because none of them had a correlation of at least 0.20 with the outcome.

Figure 8. RMSPE with All Four Types of Moderators: BART.

Note: ‘*’ identifies scenarios in which no moderator variables were offered to BART because none of them had a correlation of at least 0.20 with the outcome.

DISCUSSION

The findings from this study show that how accurately multi-site RCTs can predict impacts in individual sites depends on the variation in impacts across sites. For the four RCTs that examined the impact of educational interventions on mathematics or reading achievement, the average impact estimate provided an inaccurate prediction of the intervention’s impact for individual sites. In contrast, for the RCTs that focused on behavioral outcomes—the G.R.E.A.T. and student mentoring studies—the average impact estimate did provide fairly accurate impact predictions for individual sites. It appears that the key factor influencing the accuracy of these predictions is the extent to which the impact varies across sites: This variation was substantial for the four studies focused on mathematics and reading achievement but close to zero for the two studies focused on behavioral outcomes. Multi-site RCTs should report evidence on the cross-site impact variation to help readers assess the likelihood that the average impact reported by the RCT provides reasonably accurate impact predictions for individual sites.

When the intervention’s impact varies substantially across sites, additional findings cast doubt on whether accurate site-level impact predictions can be produced through statistical modelling. While one of the two statistical methods tested—BART—was often able to produce more accurate site-level impact predictions than the simple average of the impact estimates in the other sites, the prediction errors were still large enough to make an effective intervention appear to be ineffective or an ineffective intervention appear to be effective. The prediction errors from BART and lasso were large even with data on four types of potential impact moderators. These types of moderators are not even always available in all RCTs. And even when they are, local policymakers may not be able to predict what the values of those moderators will be locally before the intervention is implemented, making models that include them of questionable usefulness to local policymakers trying to decide whether to implement the intervention. The study’s findings suggest that even with moderators that probably could not practically be used to make local impact predictions, predictions from lasso and BART models produced impact predictions that were too inaccurate to inform local policy decisions. In addition, this study examined settings where impacts were being predicted for sites that were randomly – not systematically – different from the sites in the hypothetical studies. The ability to predict site-specific impacts would likely decline even further when considering scenarios where the sites in a study are not representative of the population of interest for predicting site-specific impacts.

However, we believe it is too early in the development of this literature to reach firm, negative conclusions about the ability to use multi-site RCTs to make accurate predictions of impact in individual sites. A more optimistic interpretation of the findings could be justified by the fact that the highest performing approach to modelling impacts—BART with rich data on impact moderators—improved the accuracy of site-level impact predictions for some of the RCTs. Furthermore, while these models are only useful to local policymakers that know or can predict the local values of those moderators in advance, there may be circumstances where local knowledge of key moderators—including the characteristics of the students who would participate and of the counterfactual condition—would allow local policymakers to benefit from these models. Assessing the practical value of these models would require research evidence on what local policymakers know or can predict in advance when deciding whether to adopt an intervention locally.

We acknowledge that the results here may be sensitive to the moderator variables that are available for use. Further theoretical or empirical research on the factors that moderate treatment effects could guide randomized trials to collect data on stronger impact moderators, and ultimately lead to better models of impact moderation that can be used to predict impacts in individual sites. Randomized trials are generally not designed to estimate site-specific impacts, and our exploration of their use for this purpose may be pushing them beyond their original goals. However, it is crucial that randomized trials do inform practice, and methods for designing trials to help facilitate their use for predicting impacts for individual sites should be encouraged.

Finally, estimates of the magnitude of prediction errors for individual sites should not be the last word on the performance of different prediction methods. The RMSPE estimates produced by this study do not account for the effects that those errors may have on policy decisions about whether to adopt interventions, or the welfare consequences of adopting ineffective interventions—or foregoing effective ones—due to errors in predicting their effects. While Orr et al. (2019) takes an initial step in that direction, additional research is needed to identify the conditions under which predictions errors of different magnitudes have serious consequences for decision-making and social welfare.