Out-of-School-Time Academic Programs to Improve School Achievement: A Community Guide Health Equity Systematic Review

Abstract

Context

Low-income and minority status in the United States are associated with poor educational outcomes, which, in turn, reduce the long-term health benefits of education.

Objective

This systematic review assessed the extent to which out-of-school-time academic (OSTA) programs for at-risk students, most of whom are from low-income and racial/ethnic minority families, can improve academic achievement. Because most OSTA programs serve low-income and ethnic/racial minority students, programs may improve health equity.

Design

Methods of the Guide to Community Preventive Services were used. An existing systematic review assessing the effects of OSTA programs on academic outcomes (Lauer et al 2006; search period 1985–2003) was supplemented with a Community Guide update (search period 2003–2011).

Main Outcome Measure

Standardized mean difference.

Results

Thirty-two studies from the existing review and 25 studies from the update were combined and stratified by program focus (ie, reading-focused, math-focused, general academic programs, and programs with minimal academic focus). Focused programs were more effective than general or minimal academic programs. Reading-focused programs were effective only for students in grades K-3. There was insufficient evidence to determine effectiveness on behavioral outcomes and longer-term academic outcomes.

Conclusions

OSTA programs, particularly focused programs, are effective in increasing academic achievement for at-risk students. Ongoing school and social environments that support learning and development may be essential to ensure the longer-term benefits of OSTA programs.

Context

In the United States, disparities in educational achievement between students from racial/ethnic minority families and those from white families, as well as between students from low-income families and those from more affluent families, are well documented.^1,2 Although reading and math scores generally have improved for all race/ethnic groups since 1992 and for all income levels since 2003, gaps in educational achievement persist.³ Disparities in student educational achievement have long-term consequences: education has been demonstrated to be one of the most important determinants of health and longevity.^4–6

Gaps in math and reading achievement expand during the summer months when regular school is not in session.⁷ The “faucet theory”^8,9 hypothesizes that summer loss is caused by the relative scarcity of academic resources for low-income students during summer when resources available during the school year are “turned off.” Higher-income students often have access to enrichment activities. “Summer loss” effects accumulate over a lifetime of schooling and are a source of the persistent achievement gap between students of lower and higher socioeconomic status (SES).^8,9 Summer out-of-school-time programs may be particularly effective in countering summer loss.

This review evaluated the effectiveness of out-of-school-time academic (OSTA) programs as a means of narrowing the academic achievement gap. A recent synthesis of prior reviews on OSTA programs calls for a new systematic review with attention to characteristics that make programs more or less effective.¹⁰ OSTA programs are defined as programs provided outside of regular school hours to students in grades K-12 who are either low-achieving or at risk of low achievement. These programs are offered during the school year—usually after school hours—or during summer recess. These programs must include an academic component, which can range from minimal academic content, such as supervised time for students to complete their homework or receive homework assistance, to more intensive tutoring or remedial classes focused on specific subjects, such as reading or math. Programs may include sports and recreation, snacks, or counseling. Attendance is most often voluntary, although students may be required to participate under certain circumstances (eg, to avoid retention in grade).

An extensive body of evidence links educational achievement and attainment to lifelong health outcomes through 3 interrelated pathways: (1) development of psychological and interpersonal strength, such as a sense of control and social support, which, in turn, contribute to healthy social interactions; (2) problem-solving abilities and the ability to pursue and maintain productive work and adequate income, and the health benefits they provide; and (3) adoption of healthy behaviors.^4,11–13 While educational experiments are few, a wide range of studies are supportive of a causal effect of education on downstream health.¹³ Standardized tests of academic achievement assess acquired knowledge and the ability to interact effectively in the classroom setting, reason, and solve problems. Because these abilities predict long-term health outcomes,^4,12,14,15 they provide a reasonable basis for use as outcomes in Community Guide health equity reviews.

Because academic problems are often associated with low family income or minority status, if effective, OSTA programs are likely to advance academic achievement of poor or minority populations. Because improved academic performance is linked to improved health status, and because poor and minority populations as a whole have lower health status, the benefits of OSTA programs may reach beyond improved academic performance to improved health equity.

In this review, focused programs were distinguished from general academic programs and from minimal academic programs. Focused programs concentrated on a single subject, such as math or reading. General academic programs focused on more than 1 subject. Minimal academic programs did not have a strong academic focus, but some included time for homework or homework assistance. Cooper’s hypothesis¹⁰ of “the congruence between program goals and program outcomes” was evaluated.

Using methods developed for the Community Guide (a program that conducts systematic reviews of public health interventions),¹⁶ this systematic review assessed the effectiveness of OSTA programs as a means to improve educational outcomes. For purposes of this review, a student population is considered at risk for low academic achievement if characterized by at least one of the following risk factors: low SES, racial/ethnic minority, low academic performance, single-parent family, low maternal education, or limited English proficiency. The plurality of poor children in the United States are low-income non-Hispanic white children (42.1% in 2010–2011) and are thus included in this review.¹⁷

Evidence Acquisition

For this review, a coordination team (the team) was constituted, including qualified systematic reviewers from the Centers for Disease Control and Prevention’s (CDC’s) Community Guide Branch, Community Preventive Services Task Force (Task Force) representatives, and subject matter experts from other CDC programs, external agencies, organizations, and academic institutions. A team of consultants with expertise on educational policies and programs was also constituted. The teams worked under the guidance of the Task Force.

Conceptual approach and analytic framework

The team hypothesized that the increased out-of-school instructional time, safe environment, enhanced socialization, and the possibility of improved nutrition provided by OSTA programs might contribute to improved cognitive performance, academic achievement, and social and emotional skills (Figure 1). Because OSTA programs may reduce at-risk students’ free out-of-school time during which juvenile crime and victimization peak, these programs may reduce delinquent behavior. However, if supervision during OSTA programs is lax, time spent in these out-of-school programs could increase deviant behavior by providing concentrated unsupervised socialization of groups of students at risk of such behavior.¹⁸

FIGURE 1.

Analytic Framework: Out-of-School-Time-Academic Programs

By providing supervised time outside of school hours, programs may increase parental work time and decrease childcare costs. The pathways described earlier and in Figure 1 illustrate how immediate outcomes could contribute to long-term improvements in educational outcomes and ultimately decreased morbidity and early mortality.^4–6,11

Research questions

The review focused on 8 research questions:

1. Are OSTA programs effective in improving academic achievement, in particular achievement in math and reading?

2. Are OSTA programs focused on specific topics, such as reading or math, more effective in improving academic achievement than programs with a more general focus? Are general programs more effective than programs with a minimal academic focus?

3. Do after-school programs and summer programs differ in effectiveness?

4. Are programs differentially effective at different grade levels?

5. Are programs with greater attendance or longer duration more effective?

6. Is OSTA tutoring more effective than group instruction?

7. Do OSTA programs have effects on nonacademic outcomes, such as delinquency and substance abuse?

8. Do OSTA effects differ for low-income or minority children versus higher-income or white children?

Methods

Search for evidence

Using Community Guide methods, the team identified a meta-analysis on OSTA by Lauer et al,¹⁹ which included studies published between January 1985 and May 2003. The meta-analysis met Community Guide standards¹⁶ and was accepted by the Task Force as the basis for this review.

To determine whether studies published after the cutoff date of the Lauer et al¹⁹ meta-analysis were consistent with the Lauer et al findings, the team conducted an update systematic search using search criteria similar to those of Lauer et al. Citations and reports published from 2003 to 2011 were searched in the following databases: ERIC, PubMed, Sociological Abstracts/Social Services Abstracts, and PsycINFO. The complete search strategy is available at www.thecommunityguide.org/healthequity/education/supportingmaterials/SS-outofschooltime.html. Reference lists of identified articles were also searched. The analysis in this review combines studies from the Lauer et al meta-analysis with more recent research.

A systematic review of summer school programs by Cooper et al,¹⁰ synthesizing studies published between 1967 and 1998, was also identified. It included 71 studies, only one of which was also included in the Lauer et al¹⁹ meta-analysis. Differences between included studies in these reviews may be a consequence of different inclusion criteria; for this reason, Cooper et al results were not included in this review.

Inclusion criteria for Community Guide update (2003–2011)

To qualify as a candidate for inclusion in this review, a study had to:

• evaluate the effectiveness of OSTA programs in improving academic achievement for students in grades K-12;

• evaluate a study population at risk of academic failure (as indicated by ≥1 of the characteristics noted earlier);

• include 1 or more outcomes: reading or math achievement as assessed through standardized test scores; high school graduation; enrollment in post–secondary education; or delinquency or substance abuse;

• have a control population or condition (treated or untreated);

• be conducted in a high-income country²⁰;

• be published in indexed scientific literature or a government document; and

• be written in English.

Studies were excluded from this review if the study population consisted exclusively of special needs or gifted students.

The Lauer et al review¹⁹ and the present update review differ in several ways: (1) the Lauer et al review included unpublished theses and dissertations, whereas this update review included only peer-reviewed published articles or government evaluations; (2) the Lauer et al review excluded studies that combined findings from multiple sites, whereas this update review included aggregated multisite studies; (3) the Lauer et al review extracted information only on reading and math outcomes; this update also assessed post–secondary academic achievement, delinquency, and substance use; and (4) whereas the Lauer et al review examined only studies conducted in the United States, this update review included studies from any high-income country.

Data abstraction and quality assessment

A full description of the process for data abstraction and quality assessment is available in Supplemental Digital Content Appendix A (available at http://links.lww.com/JPHMP/A155).

Analytic approach

The analytic approach for this review is available in Supplemental Digital Content Appendix B (available at http://links.lww.com/JPHMP/A156).

Evidence Synthesis

Study characteristics

Lauer et al¹⁹ reviewed the abstracts of 1808 citations and retrieved and reviewed 371 full-length articles, of which 35 met their inclusion criteria. The update review synthesis excluded 3 of those studies that reported only school grades,^21–23 for a total of 32 studies from Lauer et al.^24–55 The update search found 26 studies (reported in 25 publications)^56–79 that met inclusion criteria (Figure 2). By Community Guide standards,^16,80 all studies in the update were of greatest suitability of design. One⁷⁸ was excluded from analysis because of limited quality of execution. Of the remaining 25 studies, 6 (reported in 5 publications) were of good quality of execution^{56,70,75,76,79} and 19 were of fair quality.^{55,57–69,71–74,77} The combined analysis included 57 studies. Data necessary to calculate standardized mean differences (SMDs) were not available in studies assessing delinquency, drug abuse, or high school completion. Analyses were conducted in 2012–2013.

FIGURE 2. Flowchart Showing Update Search, Number of Included Studies From That Search, and Number of Included Studies From Previous Meta-analysis^a.

^aFrom Lauer et al¹⁹

All included studies were conducted in the United States, 63% in urban areas^* and S. Ross, et al (unpublished data, 1996) and the remainder in rural or mixed settings or did not report urbanicity (Table). Summer programs were evaluated in 49% of studies,^† and S. Ross, et al (unpublished data, 1996) and the remainder evaluated after-school settings. Study populations were predominantly from racial/ethnic minorities, mostly black and low-SES families. Specifically, among studies that reported race/ethnicity, 60% were majority black^‡ and S. Ross, et al (unpublished data, 1996) and among those reporting SES, 84% were majority low SES.^§ The largest proportion of programs were reading-focused^| and general academics^¶ (40% each), followed by math-focused^{29,42,49,52,54,61,77} (12%) and minimal academics^45,66,72,81 (7%); one program⁷⁷ had separate math- and reading-focused arms. Of 51 programs for which didactic approach was reported, most (47%) involved group instruction,^** 33% involved tutoring or individualized instruction,^†† and the remainder (20%) used mixed approaches.^‡‡ Four studies (in 3 articles) included controls involved in OSTA programs^64,71,77 that were less intensive or less academically rigorous than the intervention population. These studies assessed effects of programs that contained additional components.

TABLE.

Characteristics of Included Studies

Characteristic	Category		No. of Studies Reporting Characteristic (%)a (N = 57)
Setting	United States		57 (100%)
	Urbanicity	Urban	36 (63)
		Rural	5 (9)
		Mixed	8 (14)
		NR	7 (12)
Study population demographics	Grade levels served	Elementary (K-5)	28 (49)
		Elementary/middle	8 (14)
		Middle (6–8)	7 (12)
		Middle/high	3 (5)
		High (9–12)	7 (12)
		All	4 (7)
	Race/ethnicity	Majority black	25 (43)
		Majority Hispanic	4 (7)
		Majority nonwhite (unspecified)	7 (12)
		Majority white	2 (4)
		Mixed	4 (7)
		NR	15 (26)
	SES	Majority low SES	42 (74)
		<50% low SES	8 (14)
		NR	7 (12)
Intervention characteristics	Temporal locationb	Summer	28 (49)
		After-school	29 (51)
	Didactic method	Tutoring or individualized instruction	17 (30)
		Group instruction	24 (42)
		Mixed	10 (18)
		NR	6 (11)
	Program focus	Reading	23 (40)
		Math	7 (12)
		General academics	23 (40)
		Minimal academics	4 (7)

Abbreviations: NR, not reported SES, socioeconomic status.

^aPercentages may not equal 100 due to rounding.

^bThe temporal location for year-round programs is categorized by where the majority of academic instruction took place.

Intervention effects on academic achievement

Questions 1 and 2

Effectiveness of OSTA programs on math, reading, and general focus.

Reading achievement

The effects of OSTA programs on reading achievement were assessed in 45 studies^* and S. Ross, et al (unpublished data, 1996). The overall median SMD was 0.11 (interquartile interval [IQI]: 0.02–0.42). Substantial differences in effectiveness by program focus were found (Figure 3). Twenty-three evaluations^† and S. Ross, et al (unpublished data, 1996) of reading-focused programs yielded a median SMD of 0.31 (IQI: 0.02–0.58) compared with a median SMD of 0.09 (IQI: 0.00–0.26) for the 21 evaluations of general academic programs.^‡ The only minimal academic program⁴⁵ reported an SMD of 0.07 (Figure 3).

FIGURE 3. Effectiveness of OSTA Programs on Reading Achievement.

Abbreviations: IQI, interquartile interval; OSTA, Out-of-School-Time Academic; SMD, standardized mean difference.

Math achievement

Twenty-seven studies^* assessed the effects of OSTA programs on math achievement. The overall median SMD was 0.09 (IQI: −0.03 to 0.31). Six evaluations of math-focused programs^{29,42,49,52,54,77} yielded a median SMD of 0.12, compared with 20 evaluations of general academic programs^† with a median SMD of 0.065 (IQI: 0.01 to 0.24) (Figure 4). The only minimal academic program⁴⁵ reported an SMD of 0.043.

FIGURE 4. Effectiveness of OSTA Programs on Math Achievement.

Abbreviations: IQI, interquartile interval; OSTA, Out-of-School-Time Academic; SMD, standardized mean difference.

Additional stratified analyses for academic achievement

Question 3

To assess differential effectiveness by temporal setting (ie, after-school or summer programs), each level of program focus (reading-focused, math-focused, general academic, and minimal academic) was further stratified. Differential effects on reading achievement by temporal setting are small in the reading-focused stratum, as indicated by median SMDs of 0.26 (IQI: 0.0–0.50) and 0.31 (IQI: 0.02–0.89) for after-school^‡ and S. Ross, et al (unpublished data, 1996) and summer school programs,^§ respectively (Figure 5). Differential effects on reading achievement by temporal setting are larger for general academic programs, with a median SMD of 0.06 (IQI: 0.00–0.091) for after-school programs (Figure 5) compared with a median SMD of 0.20 (IQI: −0.02 to 0.38) for summer programs.^|

FIGURE 5. Effectiveness of OSTA Programs on Reading Achievement, Stratified by Temporal Setting.

Abbreviations: IQI, interquartile interval; OSTA, Out-of-School-Time Academic; SMD, standardized mean difference.

There were too few data points to draw a conclusion about the differential effects on math achievement of summer^29,52,77 versus after-school^42,49,54 math-focused programs (Figure 6). General academic programs in the summer^¶ showed larger effects on math achievement than after-school programs,^** as evidenced by the median SMDs of 0.22 (IQI: −0.05 to 0.29) and 0.04 (IQI: 0.00–0.24), respectively (Figure 6).

FIGURE 6. Effectiveness of OSTA Programs on Math Achievement, Stratified by Temporal Setting.

Abbreviations: IQI, interquartile interval; OSTA, Out-of-School-Time Academic; SMD, standardized mean difference.

Question 4

To assess differential effectiveness by student-grade level, studies were ordered by grade on the y-axis within program focus strata (Figure 7). Among the reading-focused programs, those for elementary grade students (average grade levels K-3) were effective (median SMDs of 0.44 [IQI: 0.11–1.05])^* and S. Ross, et al (unpublished data, 1996), whereas those for older elementary and middle school students (average grade levels 4–8) were not (median 0.02 [IQI: −0.06 to 0.06]).^{30,31,39,43,64,77} This relationship did not hold for general academic programs (Figure 7).

FIGURE 7. Effectiveness of OSTA Programs on Reading Achievement, Stratified by Student Grade Level.

Abbreviations: IQI, interquartile interval; OSTA, Out-of-School-Time Academic; SMD, standardized mean difference

Math-focused programs may be associated with achievement at higher-grade levels but not at lower-grade levels; however, the small number of math-focused programs limited inference (see Supplemental Digital Content Figure 8, available at: …). For general academic programs, there was no clear association between program effectiveness and student-grade level.

Question 5

Questions about program duration response effects could not be answered, because no included study reported the effects of both program duration and attendance. Although Lauer et al¹⁹ reported both floor and ceiling effects for program duration—for reading outcomes, benefit from programs with a minimum of 45 hours and no additional benefit beyond 200 hours—these findings were not corroborated in the update studies.

Question 6

Programs described as “homework assistance”^45,66,72 (some of which have minimal academic focus) and the federal Supplemental Educational Services^{55,56,60,63,67} (required to have an academic focus) were classified as tutorial programs. Programs with reading or math tutoring/individualized instruction as their main mode of didactics^† and S. Ross, et al (unpublished data, 1996) were associated with the lowest effects for both reading (median = 0.08 [IQI: 0.013–0.30] and math (median = 0.09 [IQI: 0.015–0.23]); group instruction^‡ had greater effects for both reading (median = 0.235 [IQI: 0.02–0.48]) and math (median = 0.39 [IQI: 0.09 to 0.16]); and greatest effects were associated with mixed-group and tutoring approaches^{25,36,39,40,49,50,64,74} in both reading (median = 0.375 [IQI: 0.06–0.73]) and math (effect = 0.86; 1 study).

Question 7

The small number of available studies and inconsistency of findings yielded insufficient evidence to draw conclusions on other outcomes. One study⁶⁵ reported a relative improvement of 7.3% on the Iowa Test of Basic Skills (www.riversidepublishing.com/products/itbs/), a standardized test that assesses reading, language arts, math, social studies, and science knowledge combined. Favorable effects of OSTA programs were shown for high school completion across 4 studies,^57,61,62,66 as evidenced by a median 6.8% relative change in intervention populations compared with control populations (range, −1.1% to 15.0%). Similar improvements were found for college enrollment in 3 studies,^57,61,62 with a median relative change of 7.0% (range, 2.7%–24.0%). Two studies^61,62 reported the effects of OSTA programs on college completion, one on completion of a bachelor’s degree and one on an associate’s degree; results were inconsistent, with median relative percent changes of 17.3% and −17.5%, respectively.

Mixed results were found for the effect of OSTA programs on delinquency, reported in 5 study arms from 4 studies.^68,70,72,79 The results indicated a negligible effect in the unfavorable direction: the median relative increase was 2.3% (range, −29.2% to 52.3%). The effect of OSTA programs on substance abuse also yielded inconsistent results from 4 study arms in 3 studies.^57,68,72 The median relative change of 8.8% was in the unfavorable direction (range, −33.0%, 50.0%). Overall, the small number of studies reporting these outcomes yielded insufficient evidence to draw conclusions on effectiveness.

Question 8

Few programs were reported to have a majority of higher-SES students (6 for reading programs and 4 for math programs). Comparison of effects stratified by majority low versus high SES indicated negligible differences for math programs (0.06 [IQI: −0.04 to 0.23] for low-SES students in math programs and 0.07 [IQI: −0.11 to 1.16] for higher-SES students in math programs). However, reading programs did appear to have differential effects on students from different SES backgrounds, with greater improvement among low-SES groups (0.195 [IQI: 0.02–0.43) than among higher-SES groups (−0.07 [IQI: −0.08 to 0.18]). Studies were not stratified by race/ethnicity because this characteristic is likely to be confounded by SES.

Applicability of findings

Although included studies were conducted in the United States, the team considered that the results may be applicable to other high-income nations with similar educational systems and achievement gaps. Most evaluated programs were implemented in urban settings and among low-income and racial/ethnic minority populations—predominantly black. The limited number of studies evaluating the impact of OSTA programs on academic achievement of students from rural or middle- and high-SES or predominantly white populations limits knowledge of whether such students would benefit equally from OSTA programs. The effects of OSTA programs on the academic achievement of Hispanics and racial/ethnic minority populations other than black are also unclear. The possibility of cultural and language differences suggests the modification of standard programs for Hispanics. Because most studies were implemented in elementary school settings, applicability of results to middle and high school populations is also uncertain. The results are applicable to both summer and after-school programs. Results are applicable across levels of instructional individuation, although the combination of group classes with tutoring may have greater benefits than either approach alone.

Potential harms, additional benefits, and considerations for implementation

Included studies did not assess postulated potential harms associated with OSTA programs, specifically loss of recreational time and family time. Additional benefits reported from the broader literature include more time for parents to work⁸³ and the opportunity for low-income students to receive an additional meal. Finally, participation in OSTA may reduce opportunities for part-time student employment that may provide income and promote self-confidence. However, part-time work is also associated with increased risk behavior.^84–86

Multiple implementation challenges are reported. For many federal programs, oversight is the responsibility of the state, and compliance with program requirements and enforcement are commonly incomplete.⁶⁰ School districts often do not notify parents of available free programs, such as Supplemental Educational Services; thus, programs are often underutilized.⁵⁵ In addition, participation in most OSTA programs is voluntary, and attendance may be especially low for students most in need.⁷² Inadequate staff training and high staff turnover are also reported.⁷⁷

Economic evidence

A separate systematic review assessing the economic efficiency of OSTA programs was conducted by members of the Community Guide economics team, using the same search criteria as in the effectiveness review, supplemented with economic terms and databases and standardized methods.⁸⁷ Studies of cost, cost effectiveness, and cost-benefit were assessed when available. Fourteen studies in 12 articles^{76,81,83,87–95} were included; all reported only program cost. All monetary values in this review were converted to 2012 US dollars.

Annual costs of OSTA programs ranged from $623 to $8705 per student and varied greatly by hours of operation. Eleven included studies in 9 articles^{76,81,83,89,90,92–95} provided enough information to calculate hourly cost per student, which ranged from $3.06 to $13.17. Major cost drivers included salaries for teachers and staff, costs for facilities and utilities, and transportation costs, with salaries being the largest expense. The most expensive programs were intensive, included case management (to monitor and foster the progress of individual program participants), or had more than 1 major cost driver reported. Current research does not provide sufficient data for cost-effectiveness or cost-benefit assessments.

Conclusion

Summary of findings

According to Community Guide criteria, there is strong evidence that reading-focused OSTA programs are effective in improving the reading achievement of academically at-risk students in grades K-3. There is sufficient evidence that math-focused programs are effective in improving the math achievement of at-risk students, with an indication of greater effects of math-focused programs at higher-grade levels. There is sufficient evidence of effectiveness of general academic programs in improving the reading and math achievement of academically at-risk students, although the magnitude of each effect is smaller than those from reading- and math-focused programs.

There is evidence that OSTA programs offered during the summer provide greater benefit than those offered after school, particularly general academic programs. Evidence is insufficient to determine the effectiveness of OSTA programs with minimal academic content or the effect of OSTA programs on high school completion, college enrollment, delinquency, or drug abuse.

Evidence gaps

Additional research needed to help fill gaps in knowledge about OSTA programs is detailed in see Supplemental Digital Content Appendix C (available at: http://links.lww.com/JPHMP/A157).

Discussion

This review indicated that OSTA programs overall have beneficial effects on the math and reading achievement of at-risk students. OSTA programs are not all equally effective. Academic focus (eg, on reading or math) substantially improves academic achievement. General academic programs have smaller effects, but affect achievement in more than 1 subject. This Community Guide review synthesis confirms “the congruence between program goals and program outcomes.”¹⁰

The lack of clear findings of effects of OSTA on delinquency and substance abuse may be due to the small number of studies, the harmful effects of social interaction among at-risk youth when not well supervised,¹⁸ or lack of effect.

The hypothesis that summer programs are more effective than after-school programs in improving reading and math achievement was confirmed, particularly for general academic programs. Summer programs can include more hours; after-school programs must deliver a sufficient academic dosage between the end of the regular school day and the time when students return home. Students may be tired after a full day of school and thus less receptive to further instruction. Summer programs may be particularly effective for low-income students because the academic resources available to other students during the summer are not always available for these students.^7–9 In contrast, after-school programs may be rapidly responsive to needs that arise during the school year and may occur during a greater span of the year.

Although the meta-analysis by Cooper et al¹⁰ included populations excluded in the present review, their findings were nevertheless generally consistent with this review. Cooper et al reported effects by curriculum focus and academic subject outcome separately: Comparing students exposed to a summer program either to others not exposed or to the same students prior to exposure, they found SMDs of 0.43 (95% confidence interval [CI]: 0.32–0.54) for reading programs; 0.25 (95% CI: 0.12–0.38) for combined math and reading programs (which this review classified as general programs); and 0.24 (95% CI: 0.18–0.30) for a “multiple subjects” programs (also general programs).

The limitations of this review should be recognized. Systematic reviews rely on the information provided in included studies that may lack details desired for review purposes. Descriptions of the programs themselves often lack detail so that it is difficult to determine what was done. Decisions about the classification of studies as one type or another ideally are based on available evidence, but in some cases are inferred.

Although the results of this review indicate favorable effects of OSTA programs on reading and math achievement, these programs by themselves are unlikely to bridge the achievement gap or overcome the health disparities between minority and majority children and between low-income and higher-income populations. Even when well implemented, staffed, and attended, OSTA programs are not likely to have long-term effects in the absence of educational, community, and family environments that support these benefits.^96–98 Despite the expansion of OSTA programs in recent decades, the academic achievement gaps between children from minority and majority populations, and between children from low-income and higher-income populations, persist. Even with large increases in federal No Child Left Behind funding for OSTA programs, progress in closing these achievement gaps has been slow. Nonetheless, because OSTA programs are commonly implemented in low-income communities, they could be important components of comprehensive efforts to close the achievement gap and reduce health inequities.