Does Administrative Burden Deter Young People? Evidence from Summer Jobs Programs

Abstract

Administrative burden can reduce the effectiveness of public social programs by deterring take up among adults, but we know little about the role these burdens play in public programs for young people. This paper uses empirical evidence to assess how different barriers shape adolescents’ take-up of summer jobs programs. In a Philadelphia experiment, we find that reminder emails increased application completion by 1.3 percentage points (8.8 percent (p=0.14)), with suggestively larger effects coming from reminders emphasizing short-term monetary gains. In a non-experimental analysis of Philadelphia and Chicago programs, we further show that without individualized support during enrollment, disconnected youth are less likely to participate when offered a slot than their more advantaged peers. However, offering personalized outreach and paperwork support during enrollment makes them as or more likely to participate. Given other evidence that more disconnected youth experience larger program benefits, these findings suggest administrative burden does constrain the benefits of public spending on youth programs, and that reducing burden can increase gains from social programs for young people.

Administrative burden—the logistical barriers, stigma, paperwork, cost of information acquisition, and other hassles involved in enrolling and participating in public programs—shapes individuals’ experience of government and can deter enrollment among those for whom program benefits likely outweigh the costs (Burden et al., 2012; Currie, 2006; Wichowsky and Moynihan, 2008; Herd and Moynihan, 2018). Efforts to address these barriers by reducing stigma and providing help with income validation, application completion, and remembering to apply can increase both take-up and the net benefits from these programs (Finkelstein and Notowidigdo, 2019; Bhargava and Manoli, 2015; Sommers et al., 2012; Aizer, 2007). Yet despite considerable research about overcoming barriers to social program take-up among adults, we know much less about how administrative burden affects another type of public spending: skill-building interventions for young people.

Because adolescents tend to be more sensitive to stigma and other social influences than adults, as well as more myopic (Knoll, et al., 2015; Knoll, et al., 2017; Samanez-Larkin et al., 2011), there is reason to think they may be particularly sensitive to administrative barriers surrounding enrollment in beneficial social programs. If so, adjusting recruitment and enrollment processes to reduce the impact of these barriers in social programs for young people could increase take-up or shift the composition of participants in a way that generates larger social gains.

This paper investigates two strategies that have been successful among adults—sending informational reminders and providing help with the enrollment process—in the context of summer youth employment programs (SYEPs). SYEPs are a useful setting for several reasons. First, these publicly-supported programs are widespread, offered in almost all large U.S. cities (Heller and Kessler, forthcoming). Second, there are clear benefits to SYEPs. Despite little improvement in average employment outcomes, randomized controlled trials consistently show large declines in violence and criminal justice involvement across contexts (Heller, 2014; Davis and Heller, 2020; Modestino, 2019; Gelber, et al., 2016; Valentine et al., 2017; Heller, 2021; Kessler, et al., forthcoming). Third, program benefits are heterogeneous, with the largest gains concentrated among youth facing the highest risk of negative outcomes (Heller, 2022). If this group also faces more barriers to participating, then lowering those barriers could increase the net benefits of the program. Finally, descriptive evidence suggests that despite city-wide and school-specific advertising, as well as recruitment efforts by local community providers, youth do face challenges in the application and enrollment processes. Across cities and settings, only about 15 percent of eligible youth actually apply; thousands of youth start applications without finishing them; and typically, about a quarter of those offered a slot do not actually participate.

This study uses both experimental and non-experimental variation in recruiting strategies across several SYEP settings to identify how different kinds of barriers affect both the decision to apply and the decision to participate once offered a slot. The experimental component tests two different reminder emails among 7,138 youth who began but did not complete an application for Philadelphia’s summer jobs program, called WorkReady, in the summer of 2017. Motivated by evidence that young people value earning potential and tend to be more responsive to short-term rewards (Agarwala, 2008; Tao et al., 2018; Choi and Kim, 2013; Yamashita et al., 1999; O’Donoghue and Rabin, 2001), the first reminder message emphasized the immediate financial benefits from typical summer earnings in the program (n = 2,368). The second message focused on the longer-term gains from the program, in particular the acquisition of skills useful in a future career (n = 2,369). A control group did not receive any reminder emails (n = 2,401).

Our results suggest that the email nudges increased application rates relative to the control group (by 1.3 percentage points, or 8.8 percent (p=0.14)), with messages focusing on short-run financial benefits performing suggestively better relative to control (1.8 percentage points, or 12.2 percent (p=0.09)) than those focusing on the long-run benefits (0.9 percentage points, or 6.1 percent (n.s.)). The overall efficacy of reminders, albeit modest, suggests that forgetfulness is a relevant barrier for a small proportion of youth, and that adolescents may be more responsive to strategies that emphasize the financial gains of program participation. We also explore whether the reminders change the demographic composition of the applicant pool, finding some suggestive evidence that boys may be more affected by behavioral barriers than girls.

In the non-experimental part of the paper, we shift from the application process to the take-up decision. We perform a descriptive, comparative analysis of which applicants accept their offered slot across three cohorts of SYEPs—one in Chicago and two in Philadelphia (n=4,280)—that vary in how much targeted outreach and individual support administrators use during the enrollment process. We describe those who do versus do not participate using administrative data on the demographics, school performance, criminal history, and social service use of all program applicants offered a slot after winning a randomized lottery.¹

We show that in the absence of active enrollment support like phone call reminders, personalized logistical help, and paperwork guidance, the more disconnected youth (those with higher criminal justice involvement and lower school engagement) are less likely to take up their program slot. This pattern is consistent with the idea that less advantaged youth face larger barriers to enrollment. But both across and within cities, when the recruitment process includes these additional supports, the positive selection into participation disappears. In fact, when Philadelphia added these supports between the two program years we study, selection into the program actually reversed: participants became slightly less advantaged than non-participants. Although this evidence is only descriptive, it highlights the possibility that a small amount of targeted investment during enrollment may help overcome the barriers preventing vulnerable youth from participating in social programs.

Our overall findings support two primary conclusions. First, we demonstrate that there is scope to increase youth engagement by using simple behavioral nudges that focus on overcoming inattention in the application process, perhaps more so for those that emphasize the short-term financial benefits of participation. Second, we find that an organizational focus on providing administrative support and individualized help with take-up barriers may encourage participation among the more disconnected youth who would not take up the program otherwise. Other evidence suggests that these marginal participants are likely to benefit more from participation (Heller, 2022), meaning that lowering barriers to take-up could be a cost-effective way to increase the social returns to SYEP spending. These findings suggest the administrative burdens surrounding enrollment are not acting as effective screening devices; they seem to be preventing participation among those with larger potential gains.

More broadly, our study suggests that take-up of social programming among adolescents involves some of the same barriers as social welfare programs for adults. This finding has important implications for ensuring the effective use of public spending: youth workforce development spending alone consists of over $1 billion in annual federal support, with state spending doubling in the past decade and at least $20 billion in expanded community college, apprenticeship, and other youth programs proposed in the new American Jobs Plan (Stenstrom, 2019; US Department of Labor, 2021; The White House, 2021). Continuing to explore the role of administrative burden in deterring take-up among the young people who can benefit most, as well as a better understanding of which strategies can help overcome these barriers, will help ensure the large and growing spending on youth development programming is effectively and efficiently allocated.

Enrollment Processes, Barriers, and Programs

There are multiple stages of the application and enrollment process that might prevent otherwise interested youth from participating in SYEPs. As shown in Figure 1, young people: (1) may not know about the program, understand their own eligibility to participate, or be aware of how to apply; (2) may intend to apply but forget to finish (or lack the information to complete) the application; (3) may not find out they have been accepted or may not be able to complete the relevant enrollment paperwork (including employment verification); and (4) might face logistical or family barriers that prevent them from actually participating.

Figure 1:

Application and Enrollment Process

This study focuses on barriers (2) and (3) in the context of two SYEPs: One Summer Chicago Plus (OSC+), a specialized summer jobs program for youth in Chicago at high risk of violence involvement, and WorkReady, the umbrella program for most of the City of Philadelphia’s summer jobs programming.² We use application and participation data from three cohorts of applicants: summer of 2015 for OSC+ and summers of 2017 and 2018 for WorkReady. Both programs used random lotteries in these years to allocate over-subscribed programs (impact results, as well as details on the administrative data we use in the non-experimental analysis below, are in Heller 2022). A description of both programs can be found in the Appendix.

This study begins by focusing on stage 2 barriers, using a randomized experiment in Philadelphia in 2017. The centralized online application system in Philadelphia provides some evidence that stage 2 is a bottleneck for some initially interested youth; in 2016 (the year prior to our experiment), for example, 16,000 online application forms for WorkReady were left incomplete. While it is likely that some of these unfinished applications are a result of youth needing to fix a prior mistake or starting over rather than resetting a forgotten password, the application also requires quite a bit of personal information, including parent and school information. Therefore, it is reasonable to hypothesize that forgetfulness after running into an unknown question, or a belief that program benefits do not justify the cost of finding all the relevant application information, could be preventing youth from completing the application process.³ We test the importance of these stage 2 barriers using a reminder experiment, focusing on a set of 7,138 applications that were started in the run-up to the 2017 WorkReady summer program but remained idle for at least 7 days at some stage, pre-completion.⁴

We next turn to an exploration of stage 3 barriers using a non-experimental, descriptive analysis. There is evidence of behavioral bottlenecks here as well. Among applicants offered an SYEP slot (i.e., the treatment group in the original experiments in Heller (2022)), only 47 percent participated in OSC+ (in 2015) and 50 percent in WorkReady (in 2017/18). There are many reasons youth might not participate, including finding another summer job, needing to attend summer school, providing childcare for siblings, being unable to manage transportation logistics, or not being located near the program providers offering a position. In this study, we focus on the impact of stage 3 administrative barriers by exploiting variation in the pre-program enrollment process across cities and over time (n = 1,336 for WorkReady in 2017, n = 450 for WorkReady in 2018, and n = 2,494 for OSC+).⁵ Specifically, our second set of analyses compares the demographics of those who did and did not accept SYEP slots when offered different kinds of enrollment support between program acceptance and program delivery.

Encouraging Application

Experimental Design

To assess how much inattention and forgetfulness interfere with applying to an SYEP, we conducted a randomized controlled trial on the effect of reminder emails sent to youth with partly-completed WorkReady applications. The sample consists of all youth who had incomplete online applications that had been idle for at least 7 days. We targeted this group because they may have been truly interested in the program but inhibited from completing the application due to distraction, inattention, forgetfulness, myopia, or undervaluing the future consequences of current behavior (O’Donoghue and Rabin, 2001). There is evidence to suggest that these traits may be more present in youth populations (see review in Oreopoulos et al. (2016)), and may be exacerbated by the additional stress faced by youth of less privileged backgrounds (Mullainathan and Shafir, 2013; Blair and Raver, 2016). We also used two different reminder treatments, designed to diagnose the motivations that best encourage (forgetful) youth to return to complete their applications (see Figure 2, below). The first treatment involved messaging focused on the short-run benefits from SYEP participation, namely the average financial earnings for program participants in the prior year’s program. The second treatment focused instead on longer-run benefits of the SYEP, namely connections and skills that might have career benefits.

Figure 2:

Short- and Long-Term Reminder Emails

In the online application system, individuals must provide an email address upfront to set up an account, meaning that any initiated application was immediately associated with an email address. At this point, individuals were randomly assigned to a treatment or control condition in the WorkReady system (this was not revealed to them at any point). However, only when a given application was idle (i.e., had no applicant logins) for 7 days was that applicant actually “enrolled” into our study (for context, 88.4 percent of completed applications had been submitted within a week of application initiation, so leaving an application idle for 7 days was not common among those who eventually applied). At this stage, those who had been assigned to a treatment condition ex-ante received their first reminder email, while the control subjects were not sent a reminder. Follow-up reminder emails were also sent to treatment applications every 14 days, if the application was still not complete and had again been idle for at least seven days. In total, 7,138 applicants were ultimately enrolled in our study (2,401 in control; 2,368 in short-run reminder; 2,369 in long-run reminder).

Experimental Results

Figure 3 shows the application completion rates for the treatment groups in the experiment (short-run and long-run messaging), relative to the control. These results are also shown in a regression framework in Table 1, which reports regressions of application completion on treatment assignment. We report both standard p-values using heteroskedasticity-robust standard errors, as well as p-values from randomization inference (RI). Note that because demographic characteristics are not available for all applications, no covariates are included in these regressions (see the discussion later in this section).

Figure 3:

Application Completion Rates by Treatment Status

95% confidence intervals marked by vertical bars. *p<0.1, **p<0.05, ***p<0.01

Table 1.

Application Completion Rates by Treatment Status

	(1)	(2)
Reminders (Short-Run)	0.018*
	(0.011)
Reminders (Long-Run)	0.009
	(0.010)
Reminders (Pooled)		0.013
		(0.009)
Constant (Control Mean)	0.1474	0.1474

Note: Reported coefficients from regression of application completion on indicators for treatment group (column 1) or pooled treatment indicator (column 2), with the control group omitted. Robust standard errors shown in parentheses. N = 7,138 (2,401 in control, 2,368 in short-run reminders, 2,369 for long-run reminders).

^*p<0.1,

^**p<0.05,

^***p<0.01

We find that getting some form of reminder had a modest, positive effect on application completion, raising it by 1.3 percentage points (8.8 percent) relative to control (p=0.14, RI p=0.15). While the difference between reminder versions is not statistically significant at traditional thresholds (p=0.38), the point estimate for the short-run reminder treatment is larger (a 1.8 percentage point, or 12.2 percent increase, relative to control; p=0.085, RI p=0.088). Some caution is merited in differentiating the short-run treatment arm from the control, however, since when using a free step-down resampling method (Westfall & Young 1993) to control the family-wise error rate across the two tests (control versus each individual treatment), the adjusted p-value crosses the standard 10 percent significance threshold (adjusted p=0.155). Nevertheless, the point estimates for the effect sizes here are similar to those in other nudge experiments in similar contexts (DellaVigna and Linos, 2022) and suggest that occasional reminder emails emphasizing the benefits of a program may have modest and positive effects on application completion.

Note that individual-level demographic data was collected through the WorkReady application itself, so we lack information on the demographic characteristics of the individuals who did not ultimately complete the application. The data sharing agreements for the impact experiment did not cover incomplete applications, so we are not able to link application data to other administrative data. This not only prevents sub-group analysis of treatment effects, but also eliminates our ability to control for demographics as covariates in our regressions.

That said, we are able to explore whether the demographic composition of those who do complete their applications varied by treatment group, by regressing individual demographic characteristics on treatment assignment among completed applications (n=1,116). While this is an imperfect measure, it allows us to provide some insight into how our reminder treatments may have differentially impacted subgroups. The main results for gender are shown in Figure 4, with full regression results by gender and race in Appendix Table A1. We find that the proportion of application completers who were male was 4.13 percentage points higher (p=0.192) in the treatment conditions (pooled) than in the control, which is marginally significant in the long-run treatment arm (6.1 percentage points, p=0.097). The proportion of completers who were Black or Hispanic was not meaningfully different by treatment condition.

Figure 4:

Proportion of Completed Applications from Males

95% confidence intervals marked by vertical bars. *p<0.1, **p<0.05, ***p<0.01

Given the limitations of this analysis and imprecision of these results, it is necessary to be cautious with inference here. However, a measured conclusion is that the reminder interventions may have slightly increased the proportion of male applicants. Along with a consistent pattern across SYEP studies that boys are less likely to apply to SYEPs, this finding generates a hypothesis for future research that boys may be more subject to behavioral barriers like forgetfulness than girls, and thus more receptive to nudges to apply. Additionally, existing evidence suggests that participants who would otherwise be more criminally active benefit more from SYEP participation (Heller, 2022). Since boys are disproportionately involved in crime, the potential change in gender composition may mean that the marginal applicants generate larger social benefits as well.

Encouraging Take-Up

While the email experiment provides clean identification of the effects of reminders on the initial application decision, applying is only one barrier to participation. In prior settings, even after applicants are accepted to the program, about a quarter do not end up participating. To the extent that young people fail to take up programming due to logistical barriers (stage 3 in Figure 1) rather than alternative opportunities, there may be ways for administrators to address these barriers to increase enrollment and participation. If more disconnected youth face higher barriers to participation, reducing barriers could also change the composition of participants in substantively important ways.

In the absence of experimental variation in enrollment support strategies, we turn in this section to non-experimental comparisons across cities and time. We rely on data collected about three separate cohorts of young people who were offered slots in SYEPs in Chicago (the 2015 cohort) and Philadelphia (both the 2017 and 2018 cohorts). To better understand who chooses to take up the program under different enrollment strategies, we compare the mean demographic characteristics of “participants” (applicants offered SYEP spots who end up participating for at least one day) and “non-participants” (applicants who have an SYEP slot available but do not participate) in the two cities.

Importantly, the two cities had different recruitment strategies, with Chicago providing more outreach and individualized enrollment support to accepted youth. Therefore, a between-city comparison is the first way we describe how the differences in enrollment processes correspond to differences in who chooses to take up the program when it is offered. Then, between the 2017 and 2018 Philadelphia cohorts, the WorkReady program intentionally developed and implemented a similar kind of enrollment support. This within-city difference is the second kind of variation we exploit to investigate how administrative priorities and enrollment assistance might influence who chooses to take up SYEP programs.

The youth we analyze in this section were part of a broader set of RCTs estimating the impact of SYEPs. So unlike most programs that only keep records about youth who actually take up, we are able to observe a large number of pre-program characteristics even for youth who initially applied to the program but do not end up participating. We match youth to administrative school records and arrest records from the School District of Philadelphia, Chicago Public Schools, the Philadelphia Police Department, and the Chicago Police Department. In Philadelphia, we can also match youth to other social service records using an integrated databased known as CARES, run by the city’s Data Management Office, capturing substantiated calls for child protective services, homeless shelter use, fertility, and receipt of substance abuse or mental health counseling. We focus on pre-program characteristics so that we do not confound differences in who takes up the program with the effect of the program itself.

Table 2 shows descriptive statistics by take-up decision across cities (n = 1,786 in WorkReady and 2,494 in OSC+). Comparing the participant mean to the non-participant mean columns shows how the observable characteristics of youth differ between those who accept their program slot and participate in the program versus those who do not. The “test of difference” column shows the p-value from the hypothesis test that the participant and non-participant means are equal, adjusting for randomization block.⁶

Table 2.

Descriptive Statistics by Participation Decision for Youth Offered a Program Slot

	WorkReady (2017/18 cohorts, pooled)			One Summer Chicago+
	Participant Mean	Non-Participant Mean	Test of Difference P-Value	Participant Mean	Non-Participant Mean	Test of Difference P-Value
N	896	890		1,172	1,322
Demographics
Age	15.6	15.8	<0.01	17.3	17.4	0.78
Male	0.38	0.43	0.17	0.39	0.40	0.93
Black	0.79	0.76	0.03	0.83	0.68	<0.01
Hispanic	0.10	0.14	0.02	0.15	0.29	<0.01
White	0.04	0.05	0.95	0.01	0.01	0.40
Asian	0.05	0.04	0.22
Other Race	0.02	0.01	0.51	0.01	0.02	0.41
Is a Parent	0.01	0.02	0.07
Contact with Justice System
Ever Incarcerated as a Juvenile	0.009	0.027	0.03
Ever Received Juv. Justice Service	0.015	0.024	0.24
Ever Arrested	0.032	0.046	0.17	0.240	0.212	0.33
Total Number of Prior Arrests	0.037	0.063	0.16	0.629	0.628	0.83
Violent	0.016	0.022	0.36	0.190	0.182	0.93
Property	0.011	0.026	0.13	0.105	0.091	0.61
Drug	0.000	0.004	0.10	0.059	0.074	0.57
Other	0.010	0.010	0.82	0.275	0.281	0.73
Education
Enrolled in School	0.90	0.83	<0.01	0.74	0.76	0.29
Graduated	0.05	0.11	<0.01	0.25	0.23	0.34
Grade (self-reported)	9.5	9.7	0.01	10.7	10.8	0.14
Days Absent (if enrolled)	13.5	21.3	0.00	24.4	24.1	0.79
GPA (if non-missing)	2.57	2.34	0.00	2.36	2.44	0.05
Ever Suspended (if enrolled)	0.12	0.16	0.16	0.21	0.17	<0.01
Receipt of Social Services
Ever Received Child Protection Services	0.11	0.15	0.05
Ever Stayed in a Shelter	0.03	0.03	0.90
Any Behavioral Health Service	0.23	0.27	0.16
Any Substance Abuse Services	0.01	0.02	0.03
Any Mental Health Services	0.23	0.27	0.12

Note: One WorkReady participant was missing race information. WorkReady school absence, GPA, and suspension measures reflect only youth enrolled in public, non-charter schools in the School District of Philadelphia (N=928). The School District of Philadelphia has 180 days in the school year; Chicago Public Schools has 178. The difference column shows the p-value from the test that participant and non-participant means are equal, adjusting for randomization block.

In both cities, participants are significantly more Black and less Hispanic than non-participants, suggesting that Hispanic applicants may face additional barriers to take-up. In Philadelphia, we see significant positive selection into take-up across a number of characteristics in the data from 2017 and 2018, pooled. Relative to those who did not participate, youth who take up a WorkReady slot are significantly less likely to have been previously incarcerated (0.9 versus 2.7 percent for participants versus non-participants), are less likely to be a parent (1 versus 2.4 percent), have 37 percent fewer days absent from school (13.5 versus 21.5), are less likely to have received child protection services (11 versus 15 percent), and are less likely to have received substance abuse treatment (1 versus 2 percent). In short, youth in Philadelphia who choose to participate when offered the chance are observably more advantaged than those who fail to participate.

In Chicago, on the other hand, there is much less positive selection into participation. There are no statistically significant differences between participants and non-participants in Chicago on prior arrests or days absent from school. In fact, Chicago participants are actually significantly more likely to have been suspended than non-participants (21 versus 17 percent) and have a lower GPA (2.36 versus 2.44), while in Philadelphia participants have a significantly higher GPA than non-participants (2.57 versus 2.34) and are less likely to have been suspended (though the difference, 12 versus 16 percent, is not significant).

The descriptive patterns of who takes up the program show a clear difference across cities. Participants in WorkReady are better off across many characteristics than those who decide not to participate, while participants in OSC+ are similar, or slightly less advantaged, than non-participants. Many things about program operations, as well as counterfactual opportunities, vary across cities. But a key candidate explanation for the difference is Chicago’s administrative emphasis on putting extra resources into enrolling youth at high risk of negative outcomes: multiple phone calls to households with accepted youth, advice on obtaining necessary documents and paperwork, and other personalized advice and support. The take-up patterns are consistent with the idea that this extra effort helps the more vulnerable youth overcome the barriers they face in enrolling.

In Philadelphia, the adaptations PYN made between the two study cohorts—including considerable investments in supporting enrollment among youth accepted to the program with repeated calls to inform youth about their slots, in-person help sessions that offer individual reminders about and help with paperwork completion, and individualized advice on obtaining necessary documentation—provide some additional support for the hypothesis that administrative support can increase take-up among youth with higher risk of negative outcomes. The first descriptive evidence that those efforts mattered is that across the two program years, take-up among treatment youth rose from 45 percent to 67 percent. Table 3 shows that the 2018 cohort has somewhat lower levels of justice system involvement and social service use than the 2017 cohort regardless of take-up, so some of this difference could be because the 2018 youth faced fewer barriers to participation in general. Nonetheless, the increase in take-up is consistent with the possibility that the extra help overcoming paperwork barriers mattered.

Table 3.

WorkReady Descriptive Statistics by Participation Decision for Youth Offered a Program Slot by Cohort

	2017 Cohort			2018 Cohort			Test of Difference P-Value, 2017 to 2018
	Participant Mean	Non-Participant Mean	Test of Difference P-Value	Participant Mean	Non-Participant Mean	Test of Difference P-Value
N	595	741		301	149
Demographics
Age	15.7	15.9	<0.01	15.3	15.5	0.21	0.77
Male	0.39	0.43	0.30	0.37	0.42	0.34	0.75
Black	0.77	0.76	0.10	0.82	0.76	0.12	0.53
Hispanic	0.11	0.14	0.04	0.09	0.12	0.27	0.97
White	0.06	0.06	0.99	0.00	0.01	0.65	0.82
Asian	0.06	0.04	0.71	0.03	0.07	0.14	0.22
Other Race	0.01	0.00	0.40	0.05	0.05	0.78	0.90
Is a Parent	0.02	0.03	0.15	0.00	0.01	0.16	0.87
Contact with Justice System
Ever Incarcerated as a Juvenile	0.013	0.032	0.03	0.000	0.000	.	0.03
Ever Received Juv. Justice Services	0.017	0.026	0.25	0.010	0.013	0.76	0.67
Ever Arrested	0.030	0.050	0.06	0.037	0.027	0.57	0.15
Total Number of Prior Arrests	0.037	0.070	0.10	0.037	0.027	0.57	0.11
Violent	0.015	0.024	0.27	0.017	0.013	0.79	0.38
Property	0.010	0.030	0.06	0.013	0.007	0.49	0.08
Drug	0.000	0.005	0.10	0.000	0.000	.	0.10
Other	0.012	0.011	0.80	0.007	0.007	0.99	0.87
Education
Enrolled in School	0.89	0.82	0.01	0.92	0.86	0.07	0.87
Graduated	0.07	0.11	0.04	0.03	0.10	0.02	0.31
Grade (self-reported)	9.7	9.9	0.02	8.9	9.1	0.36	0.83
Days Absent (if enrolled)	15.2	23.9	<0.01	9.6	9.3	0.83	0.00
GPA (if non-missing)	2.61	2.30	<0.01	2.49	2.53	0.79	0.05
Ever Suspended (if enrolled)	0.11	0.17	0.08	0.14	0.13	0.85	0.31
Receipt of Social Services
Ever Received Child Protection Services	0.11	0.17	0.01	0.11	0.09	0.60	0.07
Ever Stayed in a Shelter	0.04	0.03	0.52	0.02	0.03	0.31	0.25
Any Behavioral Health Service	0.23	0.29	0.04	0.23	0.19	0.38	0.06
Any Substance Abuse Services	0.01	0.02	0.05	0.00	0.01	0.32	0.32
Any Mental Health Services	0.23	0.29	0.02	0.23	0.19	0.38	0.05

Note: One WorkReady participant was missing race information. School absence, GPA, and suspension measures reflect only youth enrolled in public, non-charter schools in the School District of Philadelphia (2017 N=708, 2018 N=220). The School District of Philadelphia has 180 days in the school year. The difference column shows the p-value from the test that participant and non-participant means are equal, adjusting for randomization block.

Additionally, Table 3 shows how the selection into take-up changed across the two Philadelphia cohorts. The left-hand panel of Table 3 shows bigger and more significant differences between participants and non-participants for the 2017 cohort than in the full sample in Table 2. And in all cases with significant differences, the participants are more advantaged: they have significantly lower incarceration, arrest, and suspension rates, as well as lower receipt of child protection services and use of behavioral health services, than those who do not take-up the program. However, in 2018, with the additional support, the observable selection into take-up on these metrics actually reverses. In fact, only three of the observable differences from 2017 are still statistically significant between participants and non-participants in 2018 – school enrollment, graduation, and use of mental health services. The other differences that had significantly favored those who chose to participate in the 2017 cohort (incarceration, several arrest types, child protection services, and overall behavioral health receipt) all flip direction, with participants being at higher risk of these outcomes than non-participants in 2018.

The last column in Table 3 reports whether the changing selection into take up significantly varies across cohort. The p-value comes from regressing each characteristic on an indicator for participation, an indicator for the 2018 cohort, and the interaction of the two. (Note that we are unable to run an equivalent test across cities, since the data are held on different institutions’ servers per our legal data-sharing agreements.) The p-value for the interaction, reported in the table, tests whether the selection into take-up differs across cohorts, controlling for the overall differences in participants and applicant pools across years.

Quite a few of the differences in crime, education, and social service receipt are significant at the 0.05 or 0.1 level. In all cases, the interaction point estimate shows that the change in take-up across cohorts resulted in a more negatively selected group of participants (more incarceration, arrests, absences, child protective services, and behavioral health services). That is, the additional enrollment support corresponds to a significantly more disconnected population of participants. Given the evidence in Heller (2022) that those at higher risk of negative outcomes benefit from SYEPs more, it is likely that the marginal participants experienced bigger declines in crime and other negative outcomes. This pattern is not consistent with the idea that the administrative burdens surrounding enrollment can act as effective screening devices. Rather, they seem to be preventing participation among those with larger potential gains.

Of course, enrollment support was not the only difference between the 2017 and 2018 cohorts in Philadelphia; this is a descriptive pattern that could be confounded by other changes. Nonetheless, this within-city reversal of selection suggests that providing extra resources to alleviate the administrative burden of moving youth from accepted to participating could be an effective way to reach vulnerable youth who might not otherwise participate. Expanding the reach of programs is often an articulated goal of policymakers and program operators; it was an explicit reason PYN participated in a lottery at all, as a way to expand access to youth without existing relationships to local program providers or who faced other barriers to participation. So to the extent that policymakers want social programs to serve youth who lack existing connections to non-profit providers, centralized administrative support that diminishes how much new outreach the resource-constrained local service agencies have to do may be quite useful.

Conclusion

Using both experimental and non-experimental variation in recruitment strategies, this paper provides some empirical guidance about the “plumbing” (Duflo, 2017) of how program recruitment interacts with the barriers youth face in applying and participating in SYEPs. The experimental findings suggest that forgetfulness or inattention does contribute to youth failing to complete initiated applications. Although the effect size is modest, there appears to be some scope for increasing application rates by sending reminder emails. Consistent with other evidence that young people are more attentive to short-term gains than long-term rewards, the emails that emphasize summer earnings had the clearest impact.

The non-experimental comparison across cities and cohorts suggests that the paperwork and logistical burden of participating discourages some youth, especially those who are less connected to school and more involved in the legal and behavioral health systems. Because these same youth experience larger gains from participating, there is a high payoff to figuring out how to help them overcome the barriers to participation. In settings where program providers offered personalized outreach, reminders, and support through the enrollment process, these youth were more likely to actually participate once offered a slot. Though there are other differences across cities and cohorts that may contribute to this pattern, it is at least suggestive evidence that additional spending on centralized support through the enrollment process could help maximize the gains of social programs that benefit more disconnected youth. The results are also consistent with anecdotal evidence from program providers, who frequently discuss the challenges involved in helping their target populations obtain the required governmental identification or locate their social security numbers.

The role of enrollment support in helping disconnected youth engage in programming is an important insight for program administrators. Local providers may not have much incentive or capacity to engage in additional spending on recruitment when they can find other, more connected youth who are easier to serve. To the extent that program administrators and policymakers want to serve youth who are more involved in the criminal justice or social service systems, it is likely that they need to provide additional resources and incentive structures to accomplish that. Beyond the interventions tested here, future research could usefully explore solutions to some of the other barriers highlighted in Figure 1, such as simplifying or shortening application forms and processes, finding ways to help youth obtain government identification, and easing transportation or family constraints that may make it difficult to participate.

More broadly, the findings of this paper emphasize that the behavioral and logistical barriers that have been the subject of much research among adult safety net programs, and to some extent in the school attendance literature, are also relevant to non-school programs that aim to engage young people. A better understanding of the administrative burdens that prevent youth from engaging in socially beneficial programming, as well as which strategies help overcome them, will help policymakers maximize the gains from public spending on youth programming.

Appendix

Description of Programs and Enrollment Processes

OSC+ is a targeted program designed specifically for youth at high risk of violence involvement and run by Chicago’s Department of Family and Support Services (DFSS). They contract with local agencies to administer the program, which are selected partly for their experience working with disconnected youth in these communities. In 2015, program providers recruited applicants from 49 public high schools, chosen based on rates of community violence, with all 16- to 21-year-old students in those schools eligible to apply. To identify interested applicants, providers used their existing social networks within the relevant neighborhoods, advertised with posters within the schools, and set up recruitment tables in schools. DFSS consistently emphasized the need for providers to go above and beyond normal practice to ensure the target population applied to the program and enrolled once accepted. They tracked program take-up prior to the start of the program, motivating providers to call youth who had not yet filled out the appropriate paperwork or accepted their slot and help them solve any information or logistical problems they might face.

Program providers often had difficulty connecting with youth accepted to the program, particularly since phone calls seemed to be the most effective method of contacting youth, and the hours during which youth were available (typically 4–7pm) did not always overlap with the providers’ business hours. Additionally, the time required to call each accepted youth (or their parents and guardians), sometimes multiple times, often exceeded the capacity of program staff. To overcome this difficulty, volunteers from other organizations helped make phone calls. Callers were given a script explaining the job placement and what youth needed to do to accept the job, including reminders of the documents required. If necessary, the providers gave information on the process of acquiring documents like a birth certificate or social security card.

DFSS’s contract with the program providers specified that they were paid based on the number of program slots they filled, but that they should only fill slots with the specific youth who had won the lottery to ensure the fair allocation of slots. If the program slots could not all be filled with treatment youth, providers were eventually allowed to contact youth assigned to the control group waitlist to ensure the program was fully subscribed. We focus our analysis on just the treatment group, so that there is a well-defined population of youth who were offered a slot, among whom we can observe eventual participation.

WorkReady is a much less targeted summer jobs program, open to all Philadelphia youth aged 14–21 who are not enrolled in college. During the 2017–18 study period, the program’s administrator, the Philadelphia Youth Network (PYN), contracted with 50–60 local agencies to operate the program. All youth applied within the same online system. Some youth had pre-existing relationships with providers in the community and received assistance from them with the application. Others applied online directly to PYN, without an existing provider connection. Once applicants were accepted to the program, they had to complete a somewhat more onerous paperwork process than in OSC+, due to requirements from both federal funding sources and state law.

Importantly, the path through which youth completed their take-up process changed between the two study cohorts. In 2017, the enrollment process was left largely to local providers. In part because of the resulting low take-up rate among accepted youth in 2017, and in part due to new fingerprint requirements imposed by the state, in 2018 PYN began hosting centralized sessions to assist accepted youth with the paperwork required to participate in the program. PYN provided staff, physical space, and personal email invitations to accepted youth in support of their paperwork completion. The idea was that help with logistics like paperwork verification should increase WorkReady participation among previously unconnected youth populations, particularly in cases where local providers did not have capacity to provide this assistance directly. As in OSC+, the cost of staffing these events required recruiting some outside volunteers to help.

Effect of Reminders on Different Populations

Table A1 shows the proportion of completed applications, by treatment, that came from three demographic groups: males, Black youth, and Hispanic youth. We find that the proportion of application completers who were male was 4.13 percentage points higher (p=0.192) in the treatment conditions (pooled) than in the control, which is marginally significant in the long-run treatment arm (6.1 percentage points, p=0.097). The proportion of completers who were Black or Hispanic was not meaningfully different by treatment condition. Given the limitations of this analysis and imprecision of the results, it is necessary to be cautious with inference here. However, a measured conclusion is that the reminder interventions may slightly increase the proportion of male applicants. Along with indications in overall take-up rates that boys are less likely to apply to SYEPs, this finding generates a hypothesis for future research that boys may be more affected by behavioral barriers like forgetfulness than girls, and thus more receptive to nudges to apply.

Table A1.

Proportion of Completed Applications from Males, Black Youth, and Hispanic Youth

	Male		Black		Hispanic
	(1)	(2)	(3)	(4)	(5)	(6)
Reminders (Short-Run)	0.023		0.008		−0.008
	(0.036)		(0.025)		(0.023)
Reminders (Long-Run)	0.061*		−0.018		−0.002
	(0.037)		(0.026)		(0.023)
Reminders (Pooled)		0.041		−0.005		−0.005
		(0.032)		(0.022)		(0.020)
Constant	0.398***	0.398***	0.864***	0.864***	0.110***	0.110***
	(0.026)	(0.026)	(0.018)	(0.018)	(0.017)	(0.017)
N	1116	1116	1116	1116	1116	1116

Note: This table restricts the experimental sample in Philadelphia to only those who completed their application, then regresses an indicator variable for being male (columns 1 and 2), Black (columns 3 and 4), and Hispanic (columns 5 and 6) on the reminder treatments individually (column 1, 3, and 5) and pooled (column 2, 4, and 6). The control group is omitted, and robust standard errors are shown in parentheses.

^*p<0.1,

^**p<0.05,

^***p<0.01