Interventions to bolster benefits take-up: Assessing intensity, framing, and targeting of government outreach

Significance

Studies testing the impact of behaviorally informed government outreach on the take-up of social safety net programs have yielded mixed results. In four large-scale experiments conducted during the Covid-19 pandemic, we document that light-touch government outreach consistently increases take-up of the 2021 expanded Child Tax Credit and Economic Impact Payments among very low-income Californian families, with a high return on investment. At the same time, we find that commonly hypothesized approaches for increasing the effectiveness of government outreach—including higher-touch outreach, refined message framing, and more precise targeting—yield little to no additional benefit and some come at a much higher cost.

Abstract

Behaviorally informed “nudges” are widely used in government outreach but are often seen as too modest to address poverty at scale. In four field experiments over 2 y (n = 542,804 low-income households), we test whether more proactive communication, varying message framing, and more precise targeting can boost take-up of tax-based benefits in California above and beyond traditional light-touch approaches. Our interventions focused on extremely vulnerable households, most with no prior-year earnings, who were at risk of missing out on two crucial benefits: the 2021 expanded Child Tax Credit and pandemic-relief Economic Impact Payments. Light-touch outreach consistently increased take-up of these benefits by 0.14 to 2 percentage points—a 150% to over 500% relative increase—regardless of message, sample, timing, or modality. These light-touch approaches resulted in over $4 million disbursed, with a highly cost-effective return of $50 to over $8,000 per $1 spent. However, higher-touch proactive outreach, varying messaging, and more precise targeting yielded minimal additional benefits, with proactive outreach even showing negative returns. These findings demonstrate that light-touch outreach can effectively shift behavior among very vulnerable households in contexts with reduced compliance burdens, but also underscore an urgent need to rethink the role of higher-touch strategies in closing take-up gaps in social safety net programs.

One pervasive social challenge in lifting people out of poverty is how to ensure that those who are eligible for government assistance receive it. Although social safety net programs have positive long-term impacts on economic, health, and education outcomes (1–3), 20 to 50% of Americans do not take up programs for which they are eligible (4–7). Leaning on evidence from other domains, policymakers are increasingly turning toward behavioral approaches to increase program participation, including proactive government outreach, navigation assistance, and process simplification (8–11). Yet, experimental research examining the impact of these strategies has yielded mixed results and, often, smaller-than-anticipated effect sizes (6, 12–15).

To explain why light-touch behavioral interventions may fail to consistently deliver desired effects in these contexts, three hypotheses emerge. First, it is possible that the barriers to participation in social safety net programs are simply too high to be overcome by traditional light-touch approaches; higher-touch strategies may be required (6, 16). Accessing many programs requires overcoming a host of administrative burdens, including information barriers (e.g., understanding complicated eligibility criteria) and compliance barriers (e.g., completing lengthy and difficult application or claiming processes) (17). Burdens can be particularly challenging to navigate for the most vulnerable households and those who do not have prior experience with such programs. While some studies have shown that light-touch interventions can effectively reduce these barriers and increase take-up of social safety net programs, others have found null results—even in similar contexts (13, 15, 18). Relatedly, some studies have found positive effects from higher-touch methods that provide navigation assistance to help people access programs (14, 19, 20), but other studies have not (21–23), although study settings and samples differ substantially. This mixed evidence base underscores the need for additional research aimed at understanding the relative impact of higher-touch versus lower-touch outreach, and in what contexts and for whom these strategies are effective.

Second, when government outreach is ineffective, it is commonly hypothesized that it is because the message content itself is ineffective. In other words, perhaps we need to frame information differently (or provide different information entirely) in order to shift behavior. Extant literature documents ways in which subtle variation in framing or messaging can affect behaviors from voting and vaccine take-up to 401(k) enrollment and child support payments (16, 24–26), although more recent evidence suggests that the impact of varying theoretically driven and well-calibrated messages may be smaller than previously assumed (27–29). In the context of the social safety net, existing evidence is especially mixed: Some studies have found that the framing of light-touch outreach significantly affects take-up; others have found that it affects initial engagement but not necessarily the target behavior; and still others have found that it has no effect—even within similar programmatic contexts (13, 15, 18, 30–32).

Third, mass outreach, by definition, reaches many people who may not be eligible, interested, able, or willing to take action. Behavioral interventions will not be effective if they do not reach people who are both eligible and “movable”—those who are most likely to be responsive. Thus, it is often hypothesized that more narrowly targeting outreach to those who are eligible and movable will have a greater impact. Indeed, some evidence from other contexts suggests that improved targeting of interventions translates into larger effect sizes (33–36), although it is possible that better targeting based on eligibility simply shifts baseline take-up rates without changing effect sizes or responsiveness to interventions. In the context of the social safety net, finding people who are eligible and movable can be challenging. People who are more motivated to take action, people with fewer perceived (or real) barriers to take-up, or those with a stronger belief in the utility and impact of benefits programs (35, 37) may be more movable, but it may be difficult for governments to identify these subpopulations ex ante. Eligibility, on the other hand, is more clearly defined, but still may not be easy for government agencies to observe at scale.

We directly test each of the three hypotheses in four preregistered, large-scale field experiments (n = 542,804) in California in the context of the Covid-19 stimulus payments (Economic Impact Payments, EIPs) and the 2021 expanded federal Child Tax Credit (CTC)—a set of tax-based benefits that aimed to offset the economic consequences of the pandemic. This context is unique in three ways. First, the government went to unprecedented lengths to reduce logistical barriers to claiming these benefits. The vast majority of families received benefits automatically via the tax system. For those who did not—the very low income population that was the target in our studies—the government worked with third parties to create simplified online tools to claim available benefits. Second, under the expanded eligibility criteria, many of the poorest families were newly eligible for the CTC. Between April 2020 and March 2021, the information landscape around both benefit amounts and eligibility criteria was in constant flux—new pieces of major federal legislation and new ways to claim benefits were introduced, new benefits were added, and rules around eligibility continued to change and expand. As a result, the information barriers to claiming benefits were higher than in other contexts in which the take-up of the Earned Income Tax Credit (EITC) has been studied (e.g., 18). Third, benefit amounts were large: Across all three rounds of EIPs and the 2021 expanded CTC, eligible families with children could receive up to $3,200 per adult and $6,100 per child in benefits, depending on household income and child age. As a comparison, the average family that claims the EITC receives around $2,700 in total (38). In other words, in this context, the benefits of claiming these tax-based payments should far exceed the cost of doing so in a simple cost–benefit calculation based on a rational actor model.

This context presents a promising setting for testing our hypotheses as, arguably, outreach should be more effective at increasing take-up when benefits are large, information costs are high, and logistical barriers have been reduced. We contribute to the broader existing literature on take-up of social safety programs by examining the relative impact of light-touch versus high-touch outreach among a very low-income population for whom information was a primary barrier, conceptually replicating previously tested message framings among a hard-to-reach sample, and leveraging rich administrative data to identify and target outreach at those who are more likely to be eligible to claim benefits.

Our results cast doubt on all three hypotheses as important explanations for the failure of outreach to have larger effects in the context of the social safety net. To be sure, we find that light-touch interventions consistently and significantly influence behavior, regardless of sample, modality, and timing, with effect sizes that range from 0.14 percentage points (pp) to 2 pp, compared to no-communication comparison groups. Conservatively, we estimate that sending light-touch outreach to the full eligible population in each study would have yielded about $4.3 million in additional benefits disbursed to low-income households—and likely much more. However, higher-touch interventions were, at most, only slightly more effective than more traditional light-touch approaches, which was not nearly enough to offset their additional cost. We find no consistent effect of different behaviorally informed messages nor do we find that outreach is significantly more effective when the sample is better targeted. These results underscore both the promise and limitations of behavioral “nudges” aimed at closing take-up gaps in social safety net programs and point toward an urgent need for further research on how to design effective and scalable higher-touch interventions.

Increasing Take-Up of the Child Tax Credit and Economic Impact Payments

In response to the Covid-19 pandemic, the United States government distributed a range of stimulus payments, including an expanded CTC and three rounds of EIPs, to provide economic aid quickly to American families. These payments were disbursed through the tax system, and in most cases receipt was automatic—families who had filed a tax return in the prior year or two received them without needing to take any action (see SI Appendix for details). However, families who had not filed taxes in prior years, typically because their incomes fell below the tax filing threshold, were at risk of not receiving the benefits. Recognizing this challenge, the Internal Revenue Service (IRS) created a separate channel for these families to claim the CTC and EIPs. These families could use a new “nonfiler tool” that avoided the need for them to complete full tax returns, thus also reducing compliance hurdles associated with accessing benefits. Additionally, Code for America (CfA) partnered with The White House and the US Treasury Department to create a separate nonfiler tool, GetCTC.org, that could be used for the same purpose. CfA also developed and runs a separate platform, GetYourRefund.org, that offers a streamlined way to file regular taxes for low-income individuals. These tools were available during specific claiming periods each year, loosely aligned with IRS’s annual tax filing and extended tax filing deadlines.

From 2021 to 2022, we conducted four randomized experiments in collaboration with the California Department of Social Services aimed at increasing claiming of the CTC and EIPs (Table 1) among very low-income households. Prior research estimated that around one-quarter of Californians enrolled in CalFresh (California’s Supplemental Nutrition Assistance Program (SNAP)) or CalWORKs (California’s Temporary Assistance for Needy Families (TANF) program) were at risk of not receiving both the pandemic stimulus payments and the expanded CTC due to not filing returns (39, 40). These children and families were some of California’s most vulnerable—they tended to live in households with little to no earnings, often headed by a single adult. The average household with children could receive over $3,000 from claiming available credits at this time. Nearly 90% of households in our sample that received a refund had no wage or salary earnings in 2020 (based on data from the California Employment Development Department), meaning that these benefits had a real and tangible impact on household income.

Table 1.

Study overview

Study	Sample	Sample size	Experimental design	Outcomes	Outreach modality
Study 1 (Sept-Oct 2021)	Previous nonfilers only	144,125 households	(1) Standard message (2) Salient assistance message	(1) Return initiation via GetCTC.org (2) Return submission via GetCTC.org	Recorded voice messages
Study 2 (Nov 2021)	Previous nonfilers only	47,693 households	(1) Psychological ownership message (2) Simplified process message	(1) Return initiation via GetCTC.org (2) Return submission via GetCTC.org	Emails
Study 3 (March–April 2022)	Includes previous filers	292,984 households	(1) No-communication control (2) Psychological ownership message (3) Simplified process message	(1) Return initiation via GetYourRefund.org (2) Return submission via GetYourRefund.org (3) State tax return filing from the Franchise Tax Board	Text messages
Study 4 (Nov 2022)	Previous nonfilers only	58,002 households	(1) Passive assistance (2) Opt-in assistance (3) Opt-out assistance	(1) Return initiation via GetCTC.org (2) Return submission via GetCTC.org	(1) Text messages (2) Proactive outbound phone calls

Notes: Sample sizes reported are for the analytic samples for each study. See SI Appendix for details on post-randomization exclusion criteria.

In all four experiments, CDSS sent outreach to low-income Californians to encourage them to use a nonfiler tool or file a full tax return to claim the CTC or other available EIPs. The sample for each study was drawn from CDSS program enrollment data for CalFresh and CalWORKs and thus consists of households that were likely income-eligible to claim the expanded CTC and EIPs. These data were merged with state tax filing data from the California Franchise Tax Board (FTB) to identify adults and children who were at risk of not receiving the credit during the pandemic due to their past state tax return filing history—so-called “previous nonfilers.”^*

Study 1 and Study 2 focused on previous nonfilers only and tested the impact of light-touch outreach through recorded voice messages (“robocalls”) and emails, respectively. In each study, we also tested different message framings. In Study 1, households were randomly assigned to receive a standard recorded voice message that provided basic information about the CTC or EIPs or a salient assistance message that provided the same information, along with the phone number for a CfA-run national hotline that individuals could use to receive filing assistance from a live person. In Study 2, households were randomly assigned to receive a psychological ownership message, which emphasized that available tax credits “belong” to recipients, or a simplified process message, which emphasized that the process of claiming available tax credits had been simplified. These messages draw on related theory on administrative burden (e.g., 17) and literature documenting the impact of reducing perceived process complexity and emphasizing psychological ownership on claiming of similar tax-based benefits, such as the EITC (e.g., 13, 18).

Study 3 tested text message outreach. Similar to Study 2, households were randomly assigned to receive a text message that emphasized psychological ownership or a simplified process or were assigned to a no-communication control group (20%). In Study 3, we also expanded the sample to include previous filers in addition to nonfilers, in order to test the impact of more precise targeting. Because benefits were distributed automatically to individuals who had filed a tax return in previous years, our hypothesis was that previous nonfilers were more likely to need to take action—and should thus be more “movable”—in this context.

In Study 4, our sample focused only on previous nonfilers (like Studies 1 and 2) and included higher-touch outreach by leveraging a CDSS hotline that was staffed by approximately 35 workers who were trained to answer questions about the tax filing process and the simplified filing tool, and could connect callers to other resources such as Volunteer Income Tax Assistance (VITA) sites. Households were randomly assigned to a passive assistance group, which was sent a text message with the phone number for the tax filing assistance hotline; an opt-in assistance group, which was sent a text message that asked recipients to reply “yes” if they wanted to receive a call from the assistance hotline; or an opt-out assistance group, which was sent a text message that informed recipients that they would also receive a call from the assistance hotline, but could opt-out via text. The passive assistance group in this study resembles the salient assistance condition in Study 1 and reflects one of the most common approaches governments use to connect residents with navigation assistance. We conceptualize proactive opt-out calls as higher-touch outreach.

The outcomes for Studies 1, 2, and 4 are return initiations and submissions via GetCTC.org. Because GetCTC.org was a tool intended specifically for disbursing the expanded CTC and EIPs, we can assume that any household that had a return accepted through this filing channel received benefits. In Study 3, which was conducted during the regular tax filing season, our outcomes are return initiations and submissions via GetYourRefund.org, as well as California state tax return filing. In this paper, we focus primarily on reporting results on return submissions, but all outcomes are included in the SI Appendix.

To claim the 2021 expanded CTC and other EIPs, families only needed to file one return per household—similar to traditional tax filing. Our primary preregistered outcomes were thus household-level measures of filing behavior. We define a “household” as all adults and children that are listed on the same benefits case, based on state SNAP and TANF program enrollment data. Benefits programs and the tax system define households differently, but in practice, we find that our definition closely captures tax filing units (i.e., the group of adults and children who are claimed on the same tax return): In Studies 1 and 2, only 1.1% of returns filed through GetCTC.org (and linked to individuals in our sample) included people from multiple households, per our definition.

However, the analytical approach utilized in Studies 3 and 4 does not allow us to construct household-level outcomes (SI Appendix). As such, for ease of interpretability across the four experiments, all results reported here focus on the filing outcomes of the individual contacted through the study (the “primary contact”). These were preregistered as secondary outcomes, and this does not meaningfully affect results since 92.6% of household returns submitted in Study 1 and Study 2 included the primary contact. Results from household-level models are included in the SI Appendix.

Results

The Effect of Light-Touch Outreach.

First, we consider the effect of three different types of light-touch outreach. Results are shown in Table 2. Overall, recorded voice messages, emails, and text messages all had significant and positive impacts on behavior relative to no outreach. In Study 1, individuals that received a recorded voice message were 0.14 pp more likely to submit a return (OLS regression, SE = 0.03, P < 0.001, n = 144,139) during the week following outreach than households that did not receive a message. In Study 2, households that received an email were 1.9 pp more likely to submit a return (OLS regression, SE = 0.11, P < 0.001, n = 47,697) during the week following the first wave of emails than households that did not receive an email.

Table 2.

Effect of light-and high-touch outreach on filing outcomes

	Initiations	Submissions	State tax returns (study 3 only)
	(1)	(2)	(3)
Panel A. Study 1: Recorded voice messages
Week 1 cohort	0.0025***	0.0014***
	(0.0003)	(0.0003)
Observations	144,139	144,139
Mean for cohorts 2-4	0.0012	0.0009
Panel B. Study 2: Emails
Week 1 cohort	0.0307***	0.0194***
	(0.0013)	(0.0011)
Observations	47,697	47,697
Mean for week 2 cohort	0.0055	0.0043
Panel C. Study 3: Text messages
Pooled treatment	0.0041***	-0.0001	0.0048*
	(0.0004)	(0.0003)	(0.0021)
Observations	278,241	278,241	292,984
Mean for control	0.0014	0.0004	0.689
Panel D. Study 4: Higher-intensity approaches
Opt-in assistance	0.0013	0.0007
	(0.0019)	(0.0015)
Opt-out assistance	0.0031*	0.0011
	(0.0016)	(0.0012)
Opt-out assistance + rec’d call *	0.0048*	0.0026+
	(0.0019)	(0.0015)
Observations†	55,838	55,838
Mean for passive assistance	0.0122	0.0075

^*Average treatment effect estimates among the subsample of households assigned to the opt-out assistance group that received a proactive call as intended. These estimates come from a separate model.

^†Number of observations for the model examining the subsample of households assigned to the opt-out assistance group that received a proactive call is 45,217.

Notes: Average treatment effect estimates from models described in the Methods section. Outcomes for Study 1 (Panel A) and Study 2 (Panel B) are indicators for primary contact return initiations (column 1) and submissions (column 2) through GetCTC.org in the one week following the first wave of outreach. Outcomes for Study 3 (Panel C) are indicators for primary contact return initiations (column 1) and submissions (column 2) through GetYourRefund.org in the 4 wk following outreach, and 2021 California state tax return filing (column 3). Outcomes for Study 4 (Panel D) are indicators for primary contact return initiations (column 1) and submissions (column 2) through GetCTC.org in the 2 wk following outreach. All models include randomization stratum fixed effects. The models shown for Study 1, Study 2, and Study 3, column 3 also control for county of residence. Robust SE in parentheses. ***P < 0.001, **P < 0.01, *P < 0.05, +P < 0.10.

In Study 3, we tested text message-based outreach among both filers and nonfilers. This sample differs from Studies 1 and 2, both of which only included nonfilers. The Supplement reports results on the subsample of nonfiler households, but results do not differ meaningfully. Among the full sample, we see that individuals that received text message outreach were 0.41 pp more likely to initiate a return (OLS regression, SE = 0.04, P < 0.001, n = 278,241), but no more likely to submit a return through GetYourRefund.org than individuals in the no-communication control group (OLS regression, β = -0.01 SE = 0.03, P = 0.81). However, because this study took place during the traditional April tax-filing season, it is possible that outreach recipients chose to file their taxes through other channels, rather than using GetYourRefund.org. Thus, we also examine the impact of outreach on California state tax return filing. We find that individuals who received a text message were significantly more likely to file a state tax return than individuals who did not receive a text message (OLS regression, β = 0.48 pp, SE = 0.20, P = 0.02, n = 292,984).

The effect of emails in Study 2 was an order of magnitude larger than the recorded voice messages and text messages in Studies 1 and 3, respectively. However, it is important to note that the baseline filing rate varied across studies. Moreover, the effects of each outreach modality are not directly comparable due to the differences in samples, outcomes, and timing across studies. Despite this, in each study, we consistently find that light-touch outreach has a small but positive effect on filing behavior, with effect sizes in line with related meta-analyses on the impact of light-touch government outreach and some studies on tax-based benefits take-up (18, 41).

The Effect of Higher-Touch Outreach.

In Study 4, we examined the impact of two higher-touch outreach approaches that offered navigation assistance—opt-in assistance and opt-out assistance—compared to light-touch informational outreach. We used text messages as the mode of delivery in Study 4—both because they appeared to be more effective than recorded voice messages and because they could reach a larger population than emails.

We find that the method of offering higher-touch assistance meaningfully affects engagement. Among households assigned to the opt-in assistance group, just 72 (0.62%) responded affirmatively, indicating that they wished to receive a call from the tax filing assistance hotline, while of households assigned to the opt-out assistance group, just 172 (0.74%) opted out. This is consistent with other research that documents the power of opt-out versus opt-in approaches to enrollment (e.g., 42). Notably, only a randomly selected 52% (n = 12,171) of those assigned to the opt-out assistance group ultimately received a live phone call from the tax filing assistance hotline due to time and capacity constraints; the remainder received a recorded voice message. Of those that received a live phone call, hotline workers were able to reach and speak to people in 1,072 households (8.8%).

Results on our main outcomes are shown in Table 2. Overall, 1.2% of individuals in the passive assistance group initiated a return and 0.76% submitted a return via GetCTC.org in the outcome period. Individuals assigned to the opt-in assistance group were no more likely to initiate (OLS regression, β = 0.13 pp, SE = 0.19, P = 0.50, n = 55,838) or submit (OLS regression, β = 0.07, SE = 0.15, P = 0.64) returns through GetCTC.org than individuals assigned to the passive assistance group. Meanwhile, individuals assigned to the opt-out assistance group were significantly more likely to initiate a return (OLS regression, β = 0.31, SE = 0.16, P = 0.05), but no more likely to submit a return (OLS regression, β = 0.11, SE = 0.12, P = 0.37) than those in the passive assistance group. We also find no significant difference in the likelihood of initiating or submitting a return between the opt-in assistance and opt-out assistance groups.

As a secondary analysis, we evaluate the effect of treatment among the subgroup of households assigned to the opt-out assistance group that were called as intended (regardless of whether the call was answered). The goal of this analysis is to understand the impact of treatment—proactive outreach—in the subset of cases where it was implemented as designed. While this analysis is exploratory, given that only half of households in the opt-out assistance group were called, it offers a helpful comparison to the estimates reported above. Because the order of households to be called was randomized, there is no concern that this subgroup of households is systematically different than the overall group—it is merely a smaller sample. We find that individuals assigned to the opt-out assistance group that actually received an outreach phone call as intended were significantly more likely to initiate a return (OLS regression, β = 0.48, SE = 0.19, P = 0.01, n = 45,217) but only marginally more likely to submit a return (OLS regression, β = 0.26, SE = 0.15, P = 0.08) than households in the passive assistance group.

The Effect of Varying Message Framing.

Across all studies where we varied message framing (Studies 1 to 3), we do not see a clear or consistent impact of framing on filing behavior (Table 3). In Study 1, the salient assistance message—which included the phone number for the CfA tax-filing assistance hotline—significantly increased engagement with the hotline from a baseline of 0.06% (OLS regression, β = 0.11, SE = 0.02, P < 0.001, n = 144,139). Yet, the salient assistance message had no effect on return submissions relative to the standard message (OLS regression, β = 0.00, SE = 0.06, P = 0.94).

Table 3.

Effect of message content on filing outcomes

	Initiations	Submissions	State tax returns (study 3 only)
	(1)	(2)	(3)
Panel A. Study 1: Recorded voice messages
Salient assistance message	-0.0004	0.0000
	(0.0007)	(0.0006)
Observations	144,139	144,139
Standard message mean	0.0184	0.0132
Panel B. Study 2: Emails
Psychological ownership message	-0.0027	-0.0015
	(0.0018)	(0.0015)
Observations	47,697	47,697
Simplified process message mean	0.0404	0.0268
Panel C. Study 3: Text messages
Psychological ownership message	0.0017***	0.0005*	0.0012
	(0.0004)	(0.0002)	(0.0018)
Observations	278,241	278,241	292,984
Simplified process message mean	0.0047	0.0001	0.693

Notes: Average treatment effect estimates from models described in the Methods section. Outcomes for Study 1 (Panel A) are indicators for primary contact return initiations (column 1) and submissions (column 2) through GetCTC.org in the 4 wk following outreach. Outcomes for Study 2 (Panel B) are indicators for primary contact return initiations (column 1) and submissions (column 2) through GetCTC.org in the one week following outreach. Outcomes for Study 3 (Panel C) are indicators for primary contact return initiations (column 1) and submissions (column 2) through GetYourRefund.org in the 4 wk following outreach, and 2021 California state tax return filing (column 3). Controls described in the Methods section. Robust SE in parentheses. ***P < 0.001, **P < 0.01, *P < 0.05, +P < 0.10

Studies 2 and 3 both tested the impact of messaging that emphasized psychological ownership versus messaging that emphasized that the process of filing had been simplified. Psychological ownership messaging has been tested in similar contexts and shown to increase interest in claiming the EITC by about 20 to 128% (13). Our results, however, are mixed. In Study 2, we find that individuals who were assigned to receive the psychological ownership email were 0.15 pp less likely to submit a return (OLS regression, SE = 0.15, P = 0.29, n = 47,697) than households that were assigned to receive the simplified process email, though this difference is not statistically significant. Meanwhile, in Study 3, we find that individuals assigned to receive the psychological ownership text message were 0.05 pp more likely to submit a return through GetYourRefund.org (OLS regression, SE = 0.02, P = 0.04, n = 278,241) than individuals assigned to receive the simplified process text message. They were also 0.12 pp more likely to file a state tax return, though this difference is not statistically significant (OLS regression, SE = 0.18, P = 0.50, n = 292,984).

The Effect of Targeting.

In Study 3, we examine heterogeneity by previous nonfiler status to measure whether targeting those who are more likely to need to take action influences the impact of light-touch outreach. Because the vast majority of people in the US file taxes each year, most people—including those in our sample—did not need to take any action to claim the EIPs or CTC. However, per the IRS’s disbursement rules, previous nonfilers did not receive benefits automatically. As such, previous nonfilers should—in theory—be more responsive to light-touch informational outreach. Indeed, among those assigned to the no-communication control group in this study, 55.1% of primary contacts identified as previous nonfilers in our sample filed a 2021 state tax return compared to 73.8% of primary contacts identified as previous filers. While this is a descriptive finding, it suggests that we were able to correctly identify individuals who were less likely to file taxes and for whom outreach should have thus been more relevant.

As shown in Table 4, we find that the effect of text message outreach on filing behavior is not significantly different among previous nonfilers than previous filers (OLS regression, β = 0.35 pp, SE = 0.50, P = 0.48, n = 292,984). Examining each subsample separately, we see marginally significant effects of outreach on state tax return filing among both groups, although the magnitude of the effect among previous nonfilers is nearly twice as large as among previous filers (previous nonfilers: OLS regression, β = 0.75 pp, SE = 0.44, P = 0.09, n = 77,433; previous filers: OLS regression, β = 0.40 pp, SE = 0.23, P = 0.09, n = 215,551). We also find no significant interaction between message content and targeting.

Table 4.

Effect of targeting: Differential effect of outreach on state tax return filing, by likely nonfiler status

	Interaction	Previous Nonfilers	Previous Filers
	(1)	(2)	(3)
Panel A: Effect of pooled treatment
Pooled treatment	0.0040+	0.0075+	0.0040+
	(0.0023)	(0.0044)	(0.0023)
Previous nonfiler	-0.2061***
	(0.0044)
Pooled treatment X previous nonfiler	0.0035
	(0.0050)
Observations	292,984	77,433	215,551
Mean for control (+ previous filer)	0.744	0.551	0.739
Panel B: Effect of treatment message
Control	-0.0040	-0.0050	-0.0041
	(0.0025)	(0.0048)	(0.0025)
Psychological ownership	-0.0002	0.0049	-0.0002
	(0.0021)	(0.0039)	(0.0021)
Previous nonfiler	-0.2052***
	(0.0031)
Control X previous nonfiler	-0.0009
	(0.0054)
Psychological ownership X previous nonfiler	0.0052
	(0.0044)
Observations	292,984	77,433	215,551
Mean for control (+ previous filer)	0.748	0.556	0.743

Notes: Estimates from models described in the Methods section. Column 1 reports estimates from a model interacting treatment with previous nonfiler status; Column 2 reports estimates for the subsample of previous nonfilers; and Column 3 reports estimates for the subset of previous filers. The outcome is an indicator for filing a 2021 California state tax return among primary contacts. All models control for randomization strata and county of residence. Robust SE in parentheses. ***P < 0.001, **P < 0.01, *P < 0.05, +P < 0.10

Cost–Benefit Analysis.

The marginal cost of light-touch outreach is low: $0.04 per recorded voice message, $0.02 per text message, and $0.003 per email. In Study 1, recorded voice messages cost about $26 per additional return submitted; in Study 2, emails cost about $0.15 per additional return submitted; and in Study 3, text messages cost just over $4 per additional return submitted. This is comparable to—and in some cases, lower than—what others have found. For instance, Finkelstein and Notowidigdo (2019) (14) found that providing information about SNAP enrollment cost around $20 per additional enrollee. Because benefits were large in our setting, this also translates into a high return on investment: For every $1 spent on recorded voice messages and emails in Studies 1 and 2, respectively, roughly $58 and $8,882 in benefits were disbursed to low-income families.

However, the cost of higher-touch outreach that requires labor costs–in this case, staffing a hotline to either receive or place proactive calls—is very high, ranging from $200,000 to over $2 million per year, depending on how many staff are hired. In Study 4, factoring in the variable cost of hotline staff as well as text message outreach costs, we find that offering passive assistance cost approximately $142 per additional return submitted, while offering opt-out assistance (proactive calls) cost $1,062 per additional return submitted.

Even when including estimated fixed costs (e.g., telecommunications infrastructure, and equipment), the benefit–cost ratio in Studies 1, 2, and 3 far exceeds a ratio of 1 (SI Appendix). This is driven, in part, by how large the monetary benefits were in this context. But many social safety net programs have similarly large benefits: The average SNAP recipient receives just under $200 per month or approximately $2,172 per year (43), and TANF recipients can receive up to $500 per month, depending on the state and other eligibility requirements (44).

At the same time, the benefit–cost ratio of higher-touch outreach in Study 4—both opt-in assistance and opt-out assistance—does not clear 1 under most sets of assumptions, once fixed costs are factored in. Larger effect sizes, lower costs, or larger benefits could all influence this calculation.

Discussion

Across four large-scale experiments, we find that light-touch interventions can effectively reduce take-up gaps and, when benefits are large, bring millions of dollars to low-income households at a very low cost. Effect sizes are modest in absolute terms, ranging from 0.14 pp to 2 pp (on return submissions), but large in relative terms: These effects represent a 150% to over 500% increase over baseline return submission rates. Overall, these findings are in line with recent studies that have found that behaviorally informed outreach increases target behaviors by about 1 to 2 pp, on average. Yet, in contrast to some previous research on the social safety net (13, 18), we find that the effect of varying message framing is not significant in our studies. This could be due to differences in samples and context, as we targeted people who may not have had prior experience with the specific platforms they were asked to use, or due to differences in the specific outcome behavior measured. It is noteworthy, however, that our findings are in line with some recent megastudies that have found differences of less than 1 pp between theoretically driven and well-calibrated messages (24, 45). Partly because of the nature of the benefit, our light-touch outreach was remarkably cost effective and had a large and tangible real-world impact. There also may be positive spillovers that we cannot capture in our studies. For instance, a positive experience claiming tax-based benefits, especially for someone who is not a regular tax filer, may increase the likelihood of claiming other benefits in the future by reducing learning costs, increasing trust, or calibrating expectations of compliance or psychological costs. Under any of these scenarios, our findings may be an underestimate of the overall long-term impact of these light-touch approaches.

Under the specific conditions present in our studies, where the benefits are significant and the marginal cost of outreach is low, we find that more precise targeting may not be cost-effective. Although Study 3 documents that the effect of outreach is larger in magnitude among previous nonfilers than previous filers, this difference is neither large nor statistically significant. That said, our sample was targeted by design, focusing on households that were participating in social safety net programs and thus known to be low-income. Our findings invite further research on the trade-offs between more precise targeting—for example, using cross-agency data linkages—and less costly mass outreach.

Finally, we find that higher-touch outreach that involved staffing a hotline that could proactively reach out to residents was, at best, slightly more effective at closing take-up gaps than light-touch outreach—and was much more costly. These findings challenge the assumption that more resource-intensive interventions necessarily lead to higher impact in these contexts. Yet, the vast majority of governments offering navigation assistance are using approaches that resemble our salient assistance treatment in Study 1 or our passive assistance treatment in Study 4. Our findings demonstrate the need for additional research on how to most effectively deliver higher-touch interventions—in particular, how intensive these approaches need to be to have a meaningful impact, as well as whether, for whom, and in what contexts they are cost-effective strategies for closing take-up gaps at scale.

Our studies also have some notable limitations. First, the context of this research—which was conducted during the height of the Covid-19 pandemic—may limit the generalizability of our findings to other contexts. Our studies were conducted from September 2021 to November 2022, against a backdrop of unprecedented efforts to reduce compliance burdens to claiming benefits and a rapidly changing economic and social landscape. Because our studies built on and adapted to this changing landscape, including evolving eligibility criteria and the introduction of new and additional filing tools, we are limited in the extent to which we can directly compare effectiveness across samples, outcomes, and modalities. Yet, the fact that we see consistently positive effects from light-touch outreach across all four studies despite a wide range of other policy changes in the background is promising evidence of a true effect within this context. Conversely, we cannot rule out that other high-touch interventions would have been more effective, even in this context. Future research could conceptually replicate similar interventions under conditions that will allow for direct comparison across approaches and further exploration of heterogeneous effects, including differential effects by household structure, experience with other social safety net programs, or average take-up in a household’s broader community.

Second, while our studies contribute directly to the literature on tax-based benefits, they differ—by design—from other research that focuses on this topic. For one, they were conducted when simplified filing—a bipartisan priority, but rarely an available option—was possible and this was the first time benefits were expanded to this population. Additionally, only Study 3 was conducted during regular tax filing season. Studies 1, 2, and 4 measure behavior in “off-cycle” return initiations and submissions via GetCTC.org, a tool that was available as part of the Covid-19 response. In comparison, the most similar tax-based benefit, the EITC, is only available to households that have earned income and requires claiming by filing a traditional tax return. As such, it is not clear whether our findings would generalize to the EITC, contexts outside of tax-based benefits programs, or where the cost–benefit calculation of accessing benefits is less clearly positive. Further research is needed to test similar interventions in other programmatic contexts and with different samples.

Third, overall utilization of GetCTC.org and GetYourRefund.org was very low, which suggests that at least some percentage of the population likely used other channels to claim the expanded CTC and EIPs and influences the conclusions we can draw. On the one hand, this could mean that our findings are overestimates of the true effect of outreach if our interventions simply led people who were planning on claiming benefits through another channel to substitute across tools and use GetCTC.org or GetYourRefund.org instead. On the other hand, our findings could be underestimates of the true effect if our interventions motivated people to file through a channel other than the ones to which we directed them. Unfortunately, we are not able to directly measure claims filed through other channels. That said, we do see in Study 3 that outreach significantly and positively impacted state tax return filing, even though we found no effect on return submissions through GetYourRefund.org. This implies that our treatments motivated at least some people to file through other means.

Taken together, these findings offer insights that have immediate implications for scholars and policymakers. Nationally, millions of dollars are being invested in interventions like those tested in our field experiments, in order to better distribute resources to low-income families. The strategies tested in our experiments form the core of related playbooks released by civic tech organizations and government digital service and customer experience teams (46–48). While our studies reinforce the promise of light-touch outreach as one potential cost-effective lever through which policymakers can start to close take-up gaps in social safety net programs, they also underscore the need for further research to identify more effective and efficient approaches to fully support those at risk of missing out on critical benefits.

Methods

Preregistration.

All four experiments were preregistered on OSF. The preregistered analysis plans (PAPs) for Study 3 and Study 4 (Study 3: https://osf.io/r9vct; Study 4: https://osf.io/axyqf) were updated after randomization, but prior to conducting data analysis, to reflect a change in analytic methods due to unanticipated challenges accessing individual-level outcome data. Our original PAPs for Study 3 and Study 4 contemplated analyzing individual-level data from GetYour Refund.org and GetCTC.org, respectively. After randomization, we were unable to access these data for either study. We developed and adopted an innovative alternative strategy for analysis that permitted valid (though slightly less precise) estimates without linking the individual-level data, described in detail in the Technical Appendix. All estimates presented in this paper for studies 3 and 4 use this method, and SE and P-value reflect the precision reduction.

Ethics Approval.

All studies reported in this manuscript were approved by the California Health and Human Services Agency Committee for the Protection of Human Subjects (FWA# 00000681; IRB# 00000552); IRB project number: 2019-002. The IRB waived informed consent for all studies.

Study Design.

All four studies involved statewide outreach campaigns that were administered by CDSS. In each study, the message content was tailored based on household language preferences (English or Spanish) and whether households included children (see SI Appendix for full message text).

Study 1—Testing the impact of light-touch outreach and message variation via recorded voice messages.

Study 1 was conducted from September to October of 2021 and tested outreach aimed at encouraging likely nonfilers to claim the 2020 CTC or other EIPs that they had missed out on by November 15, which was when the nonfiler tool closed that year (49). In a stratified randomization, nonfiler households^† that only had a phone number on file with CDSS (N = 205,217) were randomly assigned with equal probability to receive one of two recorded voice messages (“robocalls”): 1) a standard message that provided basic information about the Child Tax Credit or EIPs and a link to GetCTC.org; or 2) a salient assistance message that provided the same information as the standard message, as well as the phone number for a CfA-run national hotline that individuals could use to receive filing assistance from a live person.^‡ Individuals assigned to the salient assistance message group also received a link to GetCTC.org, but their landing page was personalized to include a banner highlighting the phone number for the assistance hotline. Households were cross-randomized to one of four time cohorts, with messages sent about one week apart in late September and early October 2021.

Our primary outcomes are a) initiation and b) submission of returns via GetCTC.org. Data come from Code for America, which manages the GetCTC.org platform. First, we use the randomized variation in timing of messages to examine the effect of outreach in a one-week period, relative to none. Specifically, we compare return initiations and submissions between cohort 1 and cohorts 2, 3, and 4 in the one week after cohort 1 received outreach, but before cohort 2 received outreach. We also compare the effect of the standard message versus the salient assistance message on return initiations and submissions over a rolling 4-wk period, calculated as the 30 d following each round of outreach.

Study 2—Testing the impact of light-touch outreach and message variation via email.

Study 2 was conducted in November 2021 (just ahead of the November 15 extended filing deadline) and tested the impact of email outreach that encouraged likely nonfilers to claim the CTC or EIPs. In a stratified randomization, nonfiler households with an email address on file with CDSS (N = 47,983) were randomly assigned with equal probability to receive one of two messages: 1) a psychological ownership message, which emphasized that available tax credits “belong” to recipients, or 2) a simplified process message, which emphasized that the process of claiming available tax credits had been simplified. Both messages directed recipients to GetCTC.org. Similar to Study 1, households were cross-randomized into two timing cohorts, with emails sent about one week apart.

Our primary outcomes are a) initiation and b) submission of returns via GetCTC.org. As in Study 1, we use the randomized variation in timing of messages to examine the effect of email outreach relative to no outreach by comparing return initiations and submissions between cohort 1 and cohort 2 in the one week after cohort 1 received outreach, but before cohort 2 received outreach. We also compare the impact of the psychological ownership and simplified process emails by comparing return initiations and submissions in the one week following outreach, pooling both cohorts.

Study 3—Measuring the impact of targeting.

Study 3 was conducted in March–April 2022 and tested the impact of text message outreach, variations in language, and targeting. This study contrasts with Studies 1 and 2 in two key ways. First, the sample included both previous nonfilers and filers (per 2018 and 2019 state tax return filing data). We estimate both the overall effect of outreach, as well as the differential effect on previous nonfilers, with the prediction that they were likely to be current nonfilers as well—meaning they were more likely to need to take action to avoid missing out on the CTC and EIPs. Second, this study took place during the regular tax filing season, which meant that individuals needed to file a traditional tax return to claim their credits at this time. Thus, all messages in this study directed recipients to GetYourRefund.org through which individuals could check whether they were eligible to submit a simplified tax return.

In a stratified randomization, 20% of households were assigned to a no-communication control group (N = 58,596), while the remainder were assigned with equal probability to receive one of two text messages: a) a psychological ownership message (N = 117,195) or b) a simplified process message (N = 117,193). Message content mirrored that of the emails used in Study 2 and all households received two messages: an initial message and a reminder message, sent approximately 2 wk later. Both messages had similar content and were specific to the assigned condition.

We examine the impact of outreach on a) initiations and b) submissions of returns through GetYourRefund.org in the 4 wk following initial outreach. In addition, we also use individual-level administrative data from the FTB to examine the impact of outreach on state tax return filing (via any method) for 2021.

Study 4—Testing higher-touch outreach.

Study 4 was conducted in November 2022 just ahead of a November 15 deadline for claiming available credits that year. As part of this study, CDSS set up a hotline that was staffed by approximately 35 workers who were trained to answer questions about the tax filing process and the simplified filing tool and could connect callers to other resources such as VITA sites.

In a stratified and clustered randomization, nonfiler households were randomly assigned to one of three conditions with equal probability: 1) a passive assistance group, which was sent a text message with the phone number for the tax filing assistance hotline; 2) an opt-in assistance group, which was sent a text message that asked recipients to reply “yes” if they wanted to receive a call from the assistance hotline; and 3) an opt-out assistance group, which was sent a text message that informed recipients that they would also receive a call from the assistance hotline, but could opt-out via text. Though some studies have tested the impact of combining proactive calls with more traditional light-touch outreach (e.g., 50), our passive assistance treatment reflects the more typical approach to offering navigation assistance (e.g., 14, 15). All messages directed recipients to GetCTC.org, and we examine the impact of outreach on (a) initiations and (b) submissions of returns in the 2 wk following outreach.

Analyses.

First, we examine the effect of light-touch outreach—recorded voice messages, emails, and text messages—on filing behavior in Studies 1 to 3, relative to a comparison condition of no outreach. All outcomes are measured at the individual-level for the primary contact (the recipient of the outreach message).

In Studies 1 and 2, we estimate the average effect of light-touch outreach using the following regression specification:

$$Y_{\mathit{ig}}={\beta }_{1}cohort{1}_{\mathit{ig}}+X_{\mathit{ig}}\delta +{\epsilon }_{\mathit{ig}},$$ [1]

where $Y_{\mathit{ig}}$ reflects the outcome of interest for individual i in stratum g, ${cohort1}_{\mathit{ig}}$ is an indicator for individual i’s assignment to the first timing cohort, and $X_{\mathit{ig}}$ is a vector of controls including randomization strata and county of residence. Outcomes are return initiation and submission via GetCTC.org during the one-week period between the first cohort of outreach and later cohorts, so the latter are effectively not treated.

In Study 3, we estimate the average effect of text message outreach on primary contact return initiations and submissions using the following regression specification:

$$Y_{\mathit{ig}}={\beta }_{1}treat\_poole{d}_{\mathit{ig}}+{\delta }_g+{\epsilon }_{\mathit{ig}},$$ [2]

where $Y_{\mathit{ig}}$ reflects the outcome of interest for individual i in stratum g, ${treat\_pooled}_{\mathit{ig}}$ is an indicator for individual i’s assignment to receive either outreach message (relative to the no communication control), and ${\delta }_g$ is randomization stratum fixed effects.

Next, in Study 4, we estimate the effect of higher-touch outreach—specifically, offering opt-in assistance and opt-out assistance—on return initiations and submissions among household primary contacts using the following regression specification:

$$Y_{\mathit{ig}}={\beta }_1{\mathit{optin}}_{\mathit{ig}}+{\beta }_2{\mathit{optout}}_{\mathit{ig}}+{\delta }_g+{\epsilon }_{\mathit{ig}},$$ [3]

where $Y_{\mathit{ig}}$ reflects the outcome of interest for individual i in stratum g, ${\mathit{optin}}_{\mathit{ig}}$ is an indicator for individual i’s assignment to the opt-in assistance group, ${\mathit{optout}}_{\mathit{ig}}$ is an indicator for individual i’s assignment to the optout assistance group, and ${\delta }_g$ is randomization stratum fixed effects. The omitted treatment condition is a passive assistance text message.

To evaluate the effect of varying message framing on filing behavior in Studies 1-3, we estimate the following regression specification among primary contacts:

$$Y_{\mathit{ig}}={\beta }_1{\mathit{treat}}_{\mathit{ig}}+X_{\mathit{ig}}\delta +{\epsilon }_{\mathit{ig}},$$ [4]

where $Y_{\mathit{ig}}$ reflects the outcome of interest for individual i in stratum g, ${\mathit{treat}}_{\mathit{ig}}$ is an indicator for individual i’s assignment to receive the salient assistance message (for Study 1) or the psychological ownership message (for Study 2 and Study 3), and $X_{\mathit{ig}}$ is a vector of controls. For Studies 1 and 2, controls include time cohort, randomization strata, and county of residence. For Study 3, controls include randomization strata when the outcome is GetYourRefund.org return initiations and submissions or randomization strata and county of residence when the outcome is 2021 state tax return filing. The comparison conditions are the standard message in Study 1 and the simplified process message in Study 2 and Study 3. Again, all outcomes reported here are measured at the individual-level among primary contacts.

Finally, in Study 3, we examine whether targeting previous nonfilers (i.e., individuals who had not filed or been claimed on a filed 2018 and 2019 state tax return) increases effect sizes using the following regression specification:

$$Y_{\mathit{ig}}={\beta }_1{\mathit{treatXnonfiler}}_{\mathit{ig}}+{\beta }_2{\mathit{treat}}_{\mathit{ig}}+{\beta }_3{\mathit{nonfiler}}_{\mathit{ig}}+X_{\mathit{ig}}\delta +{\epsilon }_{\mathit{ig}},$$ [5]

where $Y_{\mathit{ig}}$ reflects the outcome of interest—2021 state tax return filing—for individual i in stratum g, ${\mathit{treat}}_{\mathit{ig}}$ is an indicator for individual i’s assignment to receive outreach, ${\mathit{nonfiler}}_{\mathit{ig}}$ is an indicator for whether individual i was a previous nonfiler, and $X_{\mathit{ig}}$ is a vector of controls including randomization strata and county of residence. Outcomes reported here are measured among primary contacts only.

Supplementary Material

Appendix 01 (PDF)

Data, Materials, and Software Availability

Analysis code for each study have been deposited in OSF (https://osf.io/98kp5/) (51). Data are legally restricted and not publicly available. Researchers wanting to replicate our analysis will need to obtain permission from FTB, CDSS, and CfA. Interested researchers should contact CDSS directly at RADDResearch@dss.ca.gov to facilitate replication efforts, given agency approval.