Delivering information about retirement saving among Hispanic women: two Facebook experiments

Abstract

We conducted two Facebook experiments (the first one during July 21–25, 2016, and the second during April 22–25, 2018) to determine what type of message related to injunctive norms is more effective in getting Hispanic women interested in learning about financial planning for retirement. We also explore how social media tools could be used in future interventions to promote retirement saving among Hispanic women. In both experiments, we found that a message centered on peer influence may be more successful than a message centered on familism in getting Hispanic women interested in learning more about financial planning for retirement. When we disaggregate our data by age and state, we find that click-through rates were higher among Hispanic women between 45 and 55 years old, and the largest numbers of impressions were among Hispanic women in California and Texas. When we disaggregate our data by device, we find that most study participants were reached through an Android smartphone.

Introduction

The lack of financial planning and saving for retirement is an important policy issue in the United States. According to the ‘Report on the Economic Well-Being of U.S. Households in 2015’, in both 2014 and 2015, 31% of non-retired respondents did not have retirement savings (Federal Reserve, 2016). A survey on retirement preparedness shows that saving for retirement is the most important financial priority for Americans, but 54% of those who have not yet retired have no idea how much they will need to save for retirement, and 74% agree that they should be doing more to prepare for retirement (Prudential, 2016).

There are significant differences among racial and ethnic groups in retirement preparedness. In 2015, while only 26% of working-age white households did not have retirement savings, 40% of blacks and 43% of Hispanics lack any retirement savings (Federal Reserve, 2016). Levels of retirement saving are also lower for blacks and Hispanics after controlling for age and income, and a person of color is less likely to have access to an employer-sponsored retirement plan (Rhee, 2013). Indeed, equal access to a retirement plan is lacking for Hispanics according to a study conducted by Prudential (2014). While 83% of the overall population have access to employer-sponsored retirement plans, only 72% of Hispanics do, and more than half of those Hispanics in Prudential’s (2014) study expressed having a poor or very poor understanding of retirement plans.¹ Qualitative studies have also shown that Hispanics have low levels of self-reported retirement preparedness and Social Security literacy (Rabinovich et al., 2016) and face significant behavioral and cultural barriers towards retirement planning and saving (Richman et al., 2008, 2015; Blanco et al., 2017).

As a result of the gender income gap, women are also less likely to be prepared for retirement. Women are more likely than men to work in part-time jobs, which makes them less likely to have access to an employer-sponsored retirement plan. While the 2012 Survey of Income and Program Participation indicates that 46% of both women and men have a retirement savings plan, women accumulate much less in their retirement accounts (Brown et al., 2016). Data from Vanguard DC retirement accounts show that the median value accumulated in retirement accounts for women in 2014 was $24,446, or 66% of the $36,875 in median retirement savings for men (Brown et al., 2016). In 2006, Hispanic females’ median retirement income was 41% of that for white males (WISER, 2008). Hispanic females’ median retirement income is also lower than that for black females (whose median retirement income is 50% of that for white males) and lower than that for Asian females (whose median retirement is 48% of that for white males). More recently, a survey of 25,435 adults in 2015 found that 34% of all females aged 35 years and older feel financially secure, but only 27% of Hispanic females feel financially secure (Mintel, 2016).

Because Hispanic women have lower levels of wealth accumulation and retirement preparedness, they are likely to face significant challenges as they age. Hispanic women also show lower levels of financial literacy than other groups (Lusardi & Mitchell, 2007). Furthermore, women are also more likely than men to admit their lack of financial knowledge, which makes them more likely than men to search for more information on this topic (Lusardi & Mitchell, 2008, 2011a, 2011b). Promoting financial planning and saving for retirement among Hispanic women can help to diminish minority and gender disparities among older populations. Targeting Hispanic females can help their spouses as well; a recent study by Prudential (2015) showed that, among married Hispanic women, 65% take the lead role in financial and retirement planning. Improving Hispanic women’s asset building and preparedness towards retirement requires not only studying the barriers they face when planning and saving for retirement, but also studying the best way to provide them with information about financial planning for retirement. Because behavioral ‘nudges’ play an important role in prompting people to save for retirement (Choi et al., 2004), studying the use of social media as a tool for motivating Hispanic women to plan and save for retirement will be beneficial in the design of future interventions. Social media can help researchers and policy-makers to reach, in a cost-efficient way, populations that they have not been able to reach otherwise.

This study explores how to provide information about financial planning for retirement to Hispanic women through social media. We conduct two Facebook experiments designed based on the tenets of behavioral economics. We seek to develop a better understanding of what message is more effective in getting Hispanic women interested in retirement saving and how social media tools can be used to provide them with information about financial planning for retirement.

The messages used in this experiment centered on financial planning and saving for retirement. We asked adult Hispanic women to learn more about free information and resources related to retirement planning and saving provided in a website created for this study (yoplaneomiretiro.com). Our experiment targeted Hispanic women who were 33–55 years old (Generation X and younger boomers) and from the four states with the largest percentages of Hispanics of Mexican origin (Arizona, California, New Mexico and Texas). We use Facebook’s advertising platform to test the performance of our messages using the following metrics: click-through, post-reaction, post-share and page likes rates. We also use Google Analytics to determine which ad brought the most visitors to our webpage.

The reason for experimenting with real-world advertising platforms to test behavioral economics concepts is twofold: (1) typical randomized controlled trials (RCTs) are very costly and usually take a significant amount of time; and (2) subjects who know they are being observed or are part of a group may behave differently or provide different responses to the researcher. While our experiment does not recreate or replace RCTs, it provides insights into how individuals respond in a real-world scenario, where we are able to measure actual behavior. Our study also provides useful information for the design of future programs and interventions to promote financial planning for retirement among Hispanic women.

We explore how women respond to two different messages related to injunctive norms (i.e., norms perceived as what most people approve or disapprove of). We conducted the first experiment during July 21–25, 2016, and the second experiment during April 22–25, 2018. We present results for both experiments. In both experiments, we found that a message centered on peer influence may be more successful than a message centered on familism in getting Hispanic women interested in learning more about financial planning for retirement. When we disaggregate our data by age and state, we find that click-through rates were higher among Hispanic women between 45 and 55 years old, and the largest numbers of impressions were among Hispanic women in California and Texas. When we disaggregate our data by device, we find that most study participants were reached through an Android smartphone.

Our paper is organized as follows: the ‘Previous research’ section presents a literature review of previous research from behavioral economics that relates directly to our experiments, particularly the importance of culture and social norms for changing behavior related to financial planning for retirement. We also discuss in this section previous research in relation to Facebook advertising. In the section following this, we present our study aims and testable hypotheses. In the subsequent section, we describe our methodology and data. In the next section, we present the results of our experiments, and the section after this provides a discussion. The final section concludes.

Previous research

Given the nature of our study, we next discuss the theoretical foundations and previous research conducted with the purpose of evaluating the impact of social norms in behavior. We also briefly discuss the existing literature on Facebook advertising.

Social norms

Theoretical and empirical evidence shows that culture and social norms influence human behavior not only in the domain of finance, but also in other domains such as health, education, energy efficiency, voting behavior and charitable giving, among many others.² While demographic and economic characteristics play an important role in shaping human behavior in the domain of finance, research has shown that behavioral factors are just as important as or even more influential than demographics and economics (Bertrand et al., 2006; Madrian, 2014; Fertig et al., 2015).

In Akerlof and Kranton’s (2000) theoretical model, identity influences human behavior and is strongly determined by culture. Their model proposes a utility function that incorporates identity as a motivation for behavior. Conformity, which specifically refers to gaining social approval and leads to improvements in self-esteem, is an important factor in shaping behavior (Cialdini & Goldstein, 2004). Descriptive, injunctive and personal norms have been proven to influence behavior. A descriptive norm is presented as what most people do, an injunctive norm is presented as what most people approve or disapprove of and a personal norm is presented as self-based standards for behavior (Schwartz & Fleishman, 1978; Cialdini et al., 1991). Because “injunctive norms specify what ought to be done” and promise “social rewards and punishments,” they tend to be more effective than descriptive and personal norms (Cialdini et al., 1991, p. 203).³

Empirical evidence on the impact of social norms on human behavior is extensive. Below, we discuss several field experiments related to financial behavior, especially those that target changes in retirement saving. Burtless (2004), who presents a good review of the literature on social norms and retirement planning, emphasizes that people make decisions about retirement based on what their peers do. Field experiments explore the role of peer effects on retirement planning and saving, such as Duflo and Saez’s (2002, 2003) work on the impact of peer effects on retirement savings decisions. In an analysis of individual data on tax-deferred retirement accounts at a large university, Duflo and Saez (2002) find that participation rates are correlated within departments and driven by social effects such as taste, background and the environment. They expanded on this work by offering an incentive to randomly selected individuals in randomly treated departments to attend a fair on retirement saving (Duflo & Saez, 2003). They found a social spillover effect, where enrollment in retirement accounts was higher in treated departments not only among randomly selected individuals, but also among their department peers.

Other studies have also shown evidence of the importance of social norms for retirement saving. Bailey et al. (2004) conducted a field experiment among 129 students and found that descriptive and injunctive norms had a strong effect on the hypothetical decision to contribute to a 401(k) account. Fertig et al. (2015) also designed an intervention to make retirement savings visible and commonplace and found that behavioral nudges are effective in promoting retirement savings in Mexico. Regarding financial literacy, which is related to retirement planning and saving, La Chance (2014), using data from the National Capability Study, showed a neighborhood effect on financial literacy, where those living in highly educated neighborhoods are more financially literate. She also found a correlation between the level of education in an individual’s ZIP code area of residence and the probability of an individual having retirement savings.

Interventions using social norms to promote retirement savings should be designed carefully. To get individuals to save for retirement, injunctive norms are more powerful than descriptive social norms (Croy et al., 2010). Nonetheless, providing information about peer behavior could result in lower levels of retirement savings when it is framed in a way that discourages individuals because of upward social comparisons (Beshears et al., 2015).

Facebook advertising

The use of social media as a recruitment tool for research purposes is relatively new. For example, Kapp et al. (2013) claim to be the first to use Facebook advertising for the recruitment of women in the United States to answer health-related questions. Targeting women in Australia, Subasinghe et al. (2016) recruited participants for a health-related study between 2011 and 2013. Both studies conclude that Facebook is a cost-effective method to recruit participants, given the fact that Facebook allows researchers to target individuals with specific characteristics.

Whitaker et al. (2017) conducted a systematic review of the use of Facebook to recruit participants in health-related studies. They identified 35 studies, where 63% of these studies took place in the United States. Only 28% of these studies used Facebook for testing health-related interventions. Using information from all of the studies included in their systematic review, Whitaker et al. (2017) conclude that Facebook is a very useful tool for engaging ‘hard-to-reach’ populations. They also found that Facebook was more effective than other social media platforms such as Twitter and Myspace.

While Facebook advertising has been used in several health-related studies, Facebook has not been used extensively for academic studies in the area of financial education, specifically those that aim at promoting retirement saving. Using the academic search engine EBSCOhost (using ‘Academic Search Complete’ so that all databases in EBSCOhost are searched) and the key terms ‘social media’ and ‘financial education’, we were unable to find any related study. Thus, our study contributes to the literature by showing the usefulness of a social media platform such as Facebook for the purpose of promoting financial education.

Study aims and testable hypotheses

Our study has three aims. First, we seek to evaluate what form of injunctive message is more effective in increasing Hispanic women’s interest in learning more about financial planning and saving for retirement. Second, we explore how social media tools can be used to reach Hispanic women. Third, we seek to learn more about the behavior in Facebook of our target population and use this information for the design of future interventions to promote financial planning and saving for retirement among Hispanic women.

Previous research suggests three main hypotheses regarding what message will be more effective at getting Hispanic women interested in learning more about retirement planning:

• Hypothesis 1: An injunctive norm based on peer influence is more effective than a standard message.

• Hypothesis 2: An injunctive norm based on familism is more effective than a standard message.

• Hypothesis 3: An injunctive norm based on peer influence is more effective than an injunctive norm based on familism.

Hypothesis 1 is based on the tenets of behavioral economics, which offers significant evidence on the importance of peer influence for human behavior (Burtless, 2004). In our analysis, we wanted to explore whether other values that are inherent to specific cultures have more power than peer influence in shaping behavior. Therefore, Hypothesis 2 is based on the idea that a specific value can influence behavior. Because of the importance of ‘familism’ among Hispanics (Sabogal et al., 1987; Vega, 1995) and the social norm of providing for the elderly (Richman et al., 2008, 2012), one could hypothesize that the message focused on the family will be the most successful in attracting attention among the target population. Family has been identified as the most important institution among Hispanics, for whom there is strong attachment, loyalty and reciprocity within the nuclear and extended family (Sabogal et al., 1987). For Hypothesis 3, we would like to test whether a message centered on peer influence is more effective in getting Hispanic women to reach out for information about retirement planning than a message centered on ‘familism’.

Methodology and data

We conducted our experiment twice; the first round took place during July 21–25, 2016, and the second round during April 22–25, 2018. Both experiments were almost identical, with only a small variation in the message of the ad focused on familism, which we discuss below, and the available budget for the second experiment was half of what we had for the first one ($2000 for the former, $1000 for the latter).

When we first ran our experiment (July 2016), we created three ads, which can be seen in Appendix A (available online), and conducted three different campaigns using the Facebook advertising platform. When we ran our experiment for the second time (April 2018), we were able to take advantage of the Facebook feature of split testing and tested the three ads shown in Appendix B. With the Facebook split-testing feature, we were able to ensure that our three ads were shown randomly to non-overlapping groups of people, which was not the case with our first experiment.

Figure 1 provides an overview of how our Facebook experiment was conducted. We targeted Spanish-speaking women because our messages were all in Spanish. Targeting Spanish-speaking Hispanic women is important since fewer online resources on retirement saving and planning are in Spanish, and lack of English proficiency appears to affect financial behavior, such as bank account ownership, among Hispanics (Blanco et al., 2018). Nonetheless, it is important to mention that when we ran our experiment on Facebook, we selected both English- and Spanish-speaking Hispanic women so that we could reach bilingual women with different levels of proficiency.

Figure 1.

Facebook experiment at a glance.

The three ads we used for the experiment we conducted in July 2016 are the following

1. Control (standard): Start to prepare for retirement today (Empieza a prepararte para tu retiro hoy).

2. Treatment 1 (injunctive norm, type A, peer effect): Many Hispanic women like you already have a plan for retirement (Muchas mujeres hispanas como tu ya tienen un plan para el retiro).

3. Treatment 2 (injunctive norm, type B, family centered): Having a plan for retirement protects me and my family (Tener un plan del retiro me protege a mi y a mi familia).

We used the same picture in all ads to ensure consistency and ability to detect differences in performance of messages based on the content of the message. We used a generic message as our control message, and we explored two forms of injunctive norms. In Treatment 1, our message, emphasizing peer influence as well as self-empowerment, suggests that having a plan for retirement is something desirable and attainable. This message affirms that other women like them already have a retirement saving plan, and it is attainable for viewers to do the same. In Treatment 2, our message, emphasizing the importance of familial relations, suggests that having a plan for retirement is something that should be done to protect one’s self and family.

For the experiment we conducted in April 2018, we modified the message in Treatment 2 to the following: Having a plan for retirement protects you and your family (Tener un plan del retiro te protege a ti y a tu familia). It can be observed that the message we used for the April 2018 experiment is in the second person, which is consistent with the person used in Treatment 1. In our first experiment conducted in July 2016, we first ran it using the first person in order for the message in Treatment 2 to sound less authoritative. Nonetheless, a long-established principle of advertising copywriting is that advertising copy in the second person (‘you’) is more effective than advertising in the first or third person (Cook, 2001; Cui & Zhao, 2014). Thus, our second experiment conducted in April 2018 was done with the aim of avoiding the confounding effect that results from using a different message with a different grammatical person.⁴

Facebook advertisements measure a variety of data. We focus our analysis on the following statistics

• Impressions: The total number of times that the message has been seen by individuals in their individual feeds.

• Click-through rate: We construct different versions of the click-through rates. Link-click is the metric defined as the number of times that individuals take action based on what the ad instructed them to do. For this experiment, we asked the audience to click a link to land on a webpage that includes a series of bookmarks to publicly available retirement information and resources. We calculate the click-through rate by dividing the number of actions (link-clicks) by the number of impressions (version 1). We also use unique link-clicks, sessions and new users in the numerator when we estimate the different versions of our click-through rate.

• Post-reaction rate: An engagement metric, where Facebook provides the number of reactions to the ads. As noted by Facebook Ads Manager, “The reactions button on an ad allows people to share different reactions to its content: Like, Love, Haha, Wow, Sad or Angry.” We calculate the post-reaction rate as the number of reactions divided by the number of impressions.

• Post-share rate: An engagement metric, where Facebook provides the number of shares of the ads, where “People can share your ads or posts on their own or friends’ timelines, in groups and on their own pages” (Facebook Ads Manager). We calculate the post-share rate as the number of shares divided by the number of impressions.

• Page likes rate: An engagement metric, where Facebook provides the number of page likes on the ads. We calculate the page likes rate as the number of shares divided by the number of impressions.

We constructed a mobile-friendly website for our experiments that provides concise information in Spanish about retirement planning and saving and other useful links (yoplaneomiretiro.com; see Appendices C and D for screenshots of the main page and ‘about’ webpages of the website we use in the two experiments). On the first page of this website, we provide three important recommendations related to retirement preparedness: (1) determine at what age they would like to retire and their needs in retirement; (2) obtain information about saving for retirement and the benefits of social security; and (3) start to plan and save for retirement as soon as possible. We also provided on our first page basic information about the most common retirement saving plans: IRA, 401(k)/403(b) and the federal government-sponsored retirement saving plan myRA. We provided six links to useful information on our mobile friendly website. We also have ‘privacy’ and ‘about’ pages on our website, which are in both English and Spanish (the main page was only in Spanish) to inform individuals about the study. Information about our website had to be modified for the April 2018 experiment since the myRA program was terminated in July 2017, and some links we used in our July 2016 experiment were not active in April 2018.

We used Google Analytics to determine which ad brought visitors to our webpage. Google Analytics reports the total number of sessions within a date range, where a session is the period of time a user is actively engaged with the website. Through Facebook tools, we are also able to evaluate how our ads performed in different subsamples (impressions and link-clicks disaggregated by state, age and device) to learn more about our target population.

To determine what message is more effective, we tested the significance of the difference between two independent proportions from the click-through, postreaction, post-share and page likes metrics. We have the following null and alternative hypotheses:

$$\begin{gathered} H_o:P_i=P_j \\ {H}_a:P_i\ne P_j \end{gathered}$$

The null hypothesis states that there is no difference between the two population proportions of message i and message j; the alternative hypothesis states that there is a significant difference between the population proportions of message i and message j. We use a two-tailed test so that we can determine whether the proportion from message i is significantly different from message j. We report in our results section the z-test and use a 5% level of significance to determine significant differences across population proportions.

Our study has a design similar to that of the field experiment on advertising content for loan offers in South Africa conducted by Bertrand et al. (2010). While they used mail to reach 53,000 former clients, we used Facebook’s advertising platform. For the July 2016 experiment, with a budget of $2000, we obtained 167,631 impressions in five days. For the April 2018 experiment, with a budget of $1000, we obtained 89,878 impressions in three days. Facebook’s tools allow for unprecedented access to demographic and psychographic segmentation.

Our target audience in Facebook was Hispanic women from the states with the largest percentages of Hispanics of Mexican origin: Arizona, California, New Mexico and Texas.⁵ We also restricted our sample to two generations, Generation X (33–44 years old) and younger boomers (45–55 years old). Because our messages were designed in Spanish, we targeted Hispanic women who speak either only Spanish or English and Spanish. Pepperdine Graduate and Professional Schools Institutional Review Board (Protocols #16–02-205 and #18–04-788) approved these experiments.

Our methods have some limitations. In Facebook experiments, individuals are not strictly selected randomly because the Facebook advertising platform uses an optimizer strategy, by which Facebook chooses individuals in the target population who are more likely to engage with the ad. For example, we observed that our ad campaigns were more successful among women aged 44–55 years (younger boomers). After running the ad for some time, Facebook will take advantage of the information collected and will post the ads more frequently among those who respond more to them, or, for our ads, among women aged 44–55 years. In our first experiment, where we did not use the split-testing feature, it can be observed that the number of impressions was much higher for the control message. Nonetheless, when we used the split-testing feature, the number of impressions was much more evenly distributed across the three messages.

While we designed the study so as to expose one individual to one message, there is a possibility that individuals were exposed to more than one message in our first experiment (this would not be the case in the second experiment since we used the split-testing feature). In fact, in our July 2016 experiment, 167,631 people saw our ads at least once (referred to as ‘reach’). If we compare the reach with the number of impressions (179,807), we find a difference of 12,176, which tells us that it is likely that participants in our study saw an ad more than one time, might have seen multiple ads or both in our first experiment. In fact, Facebook provides a frequency statistic, which tell us that individuals saw the ad 1.07 times in this experiment. For the April 2018 experiment, we found that the frequency with which individuals saw the ad was a little higher (1.23 times).

Given privacy and confidentiality concerns, Facebook did not provide statistics on whether individuals received multiple messages for our first experiment. The second Facebook experiment, however, ensured that individuals were exposed to only one message (individuals could have been exposed to the same message more than one time, as we can see from the frequency metrics).⁶ Statistics from Google Analytics indicate that 93% and 97% of those who visited our website were first-time visitors for both experiments.

Table 1 presents the metrics of interest provided by Facebook and Google Analytics, and Table 2 presents the metrics we calculated using the Facebook and Google Analytics data from both experiments. In Table 2, we can observe the calculated click-through (different versions), post-reaction, post-share and page likes rates we will use for our test on the different ads. It is important to mention that we focus our analysis on the click-through rates calculated using impressions, which is the number of times an ad is shown. The number of impressions is an objective metric that is comparable among both experiments. Reach, which is the number of people who view the ad at least once, is estimated by Facebook using sampling and modeling. Given that our experiments have different sample sizes, and that Facebook does not provide further information on their sampling and modeling strategy for calculating reach, we decided that using the number of impressions to calculate click-through rates would be more reliable, especially given that we have data from different experiments that vary in their sample sizes.

Table 1.

Metrics provided by Facebook and Google Analytics

	July 21–25, 2016, experiment				April 22–25, 2018, experiment
	Control	Treatment 1	Treatment 2	Total (average)	Control	Treatment 1	Treatment 2	Total (average)
Reach	57,472	53,952	56,207	167,631	28,951	30,744	30,183	89,878
Impressions	64,169	54,854	60,784	179,807	36,430	37,860	36,459	110,749
Frequency	1.12	1.02	1.08	(1.073)	1.26	1.23	1.21	(1.233)
Link-clicks	1473	1444	1153	4070	1070	1205	853	3128
Link-clicks (unique)	1407	1365	1082	3854	1026	1157	817	3000
Post reaction	170	208	150	528	123	139	106	368
Post share	64	76	38	178	30	44	23	97
Page likes	9	25	5	39	40	72	45	157
Cost	0.43	0.44	0.55	(0.473)	0.31	0.27	0.39	(0.323)
Sessionsa	1172	1200	836	3208	993	1131	809	2933
New usersa	1112	1121	765	2998	964	1094	775	2833

^aMetric provided by Google Analytics; all the other metrics provided by Facebook. Totals shown in most cases, unless denoted with parentheses, which are averages.

Table 2.

Calculated metrics using data from Facebook and Google Analytics

	July 21–25, 2016, experiment				April 22–25, 2018, experiment
	Control	Treatment 1	Treatment 2	Average	Control	Treatment 1	Treatment 2	Average
Reach/impressions	0.896	0.984	0.925	0.935	0.795	0.812	0.828	0.812
Link-clicks (unique)/link-clicks	95.520	94.530	93.840	94.630	95.890	96.020	95.780	95.897
Sessions/link-click	0.796	0.831	0.725	0.784	0.930	0.940	0.950	0.940
New user/sessions	0.949	0.934	0.915	0.933	0.970	0.970	0.960	0.967
Click-through rate version 1	2.296	2.632	1.897	2.275	2.937	3.183	2.340	2.820
Click-through rate version 2	2.193	2.488	1.780	2.154	2.816	3.056	2.241	2.704
Click-through rate version 3	1.826	2.188	1.375	1.796	2.726	2.987	2.219	2.644
Click-through rate version 4	1.733	2.044	1.259	1.678	2.646	2.890	2.126	2.554
Post-reaction rate	0.265	0.379	0.247	0.297	0.338	0.367	0.291	0.332
Post-share rate	0.100	0.139	0.063	0.100	0.082	0.116	0.063	0.087
Page likes rate	0.014	0.046	0.008	0.023	0.110	0.190	0.123	0.141

Notes: Metrics calculated using data from Facebook and Google Analytics shown in Table 1. Rates are estimated in the following way:

Click-through rate version 1 = (# link-clicks/# impressions) × 100

Click-through rate version 2 = (# link-clicks unique/# impressions) × 100

Click-through rate version 3 = (# sessions/# impressions) × 100

Click-through rate version 4 = (# new users/# impressions) × 100

Post-reaction rate = (# post reactions/# impressions) × 100

Post-share rate = (# post shares/# impressions) × 100

Page likes rate = (# page likes/# impressions) × 100

Table 2 presents the constructed metrics of interest using data provided by Facebook and Google Analytics. In Table 2, we can observe that the percentage of link-clicks that are unique among the total number of link-clicks is in the range of 94–96% for both experiments. If we divide the number of sessions per link-click, we can observe that this number was much higher for the second experiment (0.94 versus 0.78), which tells us that in our second experiment, we were more successful in getting people to actually go to our website after clicking on the ad in Facebook. We are unsure why there is this difference between experiments, but it might have been related to Facebook features, our website design, the user’s browser not accepting cookies or other technical reasons. We used the same website for both experiments, so getting more people who clicked on the ad in Facebook to visit our website during the second experiment was not the result of our website design. We are unable to disentangle the major reason that precluded a larger percentage of Facebook users who clicked on our ad from actually landing on our website in the first experiment. The percentages of new users from the total number of sessions were 93% and 97% in July 2016 and April 2018, respectively.

Results

We conduct a two-sided test to determine whether the difference in the proportions is significantly different between message i and message j, and we report the difference in the proportion, the z-value for the significance of the difference in the proportion and the probability.⁷ Tables 3 and 4 present tests of significance for the difference between messages using the different metrics for the July 2016 and April 2018 experiment, respectively. In both tables, we also denote with an asterisk whether the difference is significant with a statistical power of 80%.⁸ We focus our analysis on the rates (expressed as proportions) as the metric divided by the number of impressions to determine what message was more effective. We calculate the click-through, post-reaction, post-share and page likes rates for each message and show them in the different panels of Tables 3 and 4. We next discuss our results from the different experiments.

Table 3.

Tests of significance of the difference between messages, July 21–25, 2016, experiment

	Proportion		Control – Treatment 1	Control – Treatment 2	Treatment 1 – Treatment 2
Panel A – Click-through rates and test of differences in rates, version 1 (using link-clicks)
Control	0.02296	Difference	−0.00336*	0.00399*	0.00736*
Treatment 1	0.02632	z-value	−3.7370	4.9143	8.4236
Treatment 2	0.01897	Probability	0.0002	0.0000	0.0000
Panel B – Click-through rates and test of differences in rates, version 2 (using link-clicks-unique)
Control	0.02193	Difference	−0.00295*	0.00413*	0.00708*
Treatment 1	0.02488	z-value	−3.3636	5.2221	8.3538
Treatment 2	0.01780	Probability	0.0008	0.0000	0.0000
Panel C – Click-through rates and test of differences in rates, version 3 (using sessions)
Control	0.01826	Difference	−0.00362*	0.00451*	0.00813*
Treatment 1	0.02188	z-value	−4.4545	6.3376	10.4969
Treatment 2	0.01375	Probability	0.0000	0.0000	0.0000
Panel D – Click-through rates and test of differences in rates, version 4 (using new users)
Control	0.01733	Difference	−0.00311*	0.00474*	0.00785*
Treatment 1	0.02044	z-value	−3.9416	6.8842	10.5224
Treatment 2	0.01259	Probability	0.0001	0.0000	0.0000
Panel E – Post-reaction rates and test of differences in rates
Control	0.00265	Difference	−0.00114*	0.00018	0.00132*
Treatment 1	0.00379	z-value	−3.4846	0.6291	4.0345
Treatment 2	0.00247	Probability	0.0005	0.5293	0.0001
Panel F – Post-share rates and test of differences in rates
Control	0.00100	Difference	−0.00039	0.00037	0.00076
Treatment 1	0.00139	z-value	−1.9538	2.2838	4.1025
Treatment 2	0.00063	Probability	0.0507	0.0224	0.0000
Panel G – Page likes rates and test of differences in rates
Control	0.00014	Difference	−0.00032*	0.00006	0.00038
Treatment 1	0.00046	z-value	−3.2461	1.0071	4.0003
Treatment 2	0.00008	Probability	0.0012	0.3139	0.0001

Notes: We conducted a test of the significance of the difference between two independent proportions (rates shown in Table 2 are expressed as proportions here). We report the z-values and probabilities on the significance of the difference between the rates for message i and message j.

^*Difference was significant with a statistical power of 80%.

Table 4.

Tests of significance of the difference between messages, April 22–25, 2018, experiment

	Proportion		Control – Treatment 1	Control – Treatment 2	Treatment 1 – Treatment 2
Panel A – Click-through rates and test of differences in rates, version 1 (using link-clicks)
Control	0.02937	Difference	−0.00246	0.00597*	0.00843*
Treatment 1	0.03183	z-value	−1.9454	5.0282	7.0012
Treatment 2	0.02340	Probability	0.0517	0.0000	0.0000
Panel B – Click-through rates and test of differences in rates, version 2 (using link-clicks-unique)
Control	0.02816	Difference	−0.00240	0.00575*	0.00815*
Treatment 1	0.03056	z-value	−1.9364	4.9443	6.9074
Treatment 2	0.02241	Probability	0.0528	0.0000	0.0000
Panel C – Click-through rates and test of differences in rates, version 3 (using sessions)
Control	0.02726	Difference	−0.00261	0.00507*	0.00768*
Treatment 1	0.02987	z-value	−2.1340	4.4074	6.5646
Treatment 2	0.02219	Probability	0.0328	0.0000	0.0000
Panel D – Click-through rates and test of differences in rates, version 4 (using new users)
Control	0.02646	Difference	−0.00244	0.00520*	0.00764*
Treatment 1	0.02890	z-value	−2.0257	4.5996	6.6494
Treatment 2	0.02126	Probability	0.0428	0.0000	0.0000
Panel E – Post-reaction rates and test of differences in rates
Control	0.00338	Difference	−0.00029	0.00047	0.00076
Treatment 1	0.00367	z-value	−0.6665	1.1331	1.8068
Treatment 2	0.00291	Probability	0.5051	0.2572	0.0708
Panel F – Post-share rates and test of differences in rates
Control	0.00082	Difference	−0.00034	0.00019	0.00053*
Treatment 1	0.00116	z-value	−1.4707	0.9529	2.4088
Treatment 2	0.00063	Probability	0.1414	0.3406	0.0160
Panel G – Page likes rates and test of differences in rates
Control	0.00110	Difference	−0.00080*	0.00013	0.00067*
Treatment 1	0.00190	z-value	−2.8094	−0.5144	2.3053
Treatment 2	0.00123	Probability	0.0050	0.6069	0.0211

Notes: We conducted a test of the significance of the difference between two independent proportions (rates shown in Table 2 are expressed as proportions here). We report the z-values and probabilities on the significance of the difference between the rates for message i and message j.

^*Difference was significant with a statistical power of 80%.

July 2016 experiment

Panel A in Table 3 presents the click-through rates calculated using the link-click metric provided by Facebook for the three messages. We find that the Treatment 1 ad has a higher click-through rate than the Control ad, and the difference is statistically significant (p < 0.01). We also find that the Control ad and Treatment 1 ad have a higher click-through rate than the Treatment 2 ad and that the difference is statistically significant as well (p < 0.01). The results shown in Panel B, which present the click-through rates calculated using the unique link-click metric provided by Facebook for the three messages, are very similar to those in Panel A, where the Treatment 1 ad has a higher click-through rate than the Control ad, and the Treatment 1 and Control ads have higher click-through rates than the Treatment 2 ad (difference statistically significant at p < 0.01).

We use Google Analytics data and construct click-through rates using sessions (Panel C) and new users (Panel D) instead of link-clicks from Facebook and find similar results. We find that the click-through rate for the Treatment 1 ad is higher, by a statistically significant amount (p < 0.01), than that for the Control and Treatment 2 ads. We find this test more reliable than the previous one using Facebook link-clicks because Google Analytics provides us with more accurate information about the number of users who actually visited our website. Even if Facebook reports link-clicks, visitors could get distracted and never visit our website even if they click our ads in Facebook.

Regarding engagement, we show the tests on the difference in post-reaction, post-share and page likes rates between the three ads in Panels E, F and G in Table 3. We find that the post-reaction rate for the Treatment 1 ad is higher by a statistically significant (p < 0.01) amount than for the Control and Treatment 2 ads (Panel E). For the post-share rate, we find that the Treatment 1 ad has a higher post-share rate, but the difference between this ad and the Control ad is not statistically significant (p = 0.0507, Panel E). The post-share rate for the Treatment 1 ad is higher by a statistically significant amount (p < 0.01) compared with that for the Treatment 2 ad. We also observe that the Treatment 1 ad page likes rate is higher than the rates for the Control and Treatment 2 ads, and this difference is statistically significant (p < 0.01).

April 2018 experiment

We conduct the same test for the different rates using data from our second experiment, and the results are shown in Table 4. The results in Panel A show that the Treatment 1 ad has a higher click-through rate than the Control ad, but the difference is not statistically significant (p = 0.0517). We also find that the Control and Treatment 1 ads have higher click-through rates than the rate for the Treatment 2 ad, and that the difference is statistically significant as well (p < 0.01). The results in Panel B are similar to those shown in Panel A. Using unique link-clicks to calculate the click-through rate, the Treatment 1 ad has a higher click-through rate than the Control ad, but the difference is not statistically significant (p = 0.0528). With this metric, we also find that the Control and Treatment 1 ads have a higher click-through rate than the Treatment 2 ad and that the difference is statistically significant as well (p < 0.01).

When we construct click-through rates using sessions and new users, we find different results from those found using Facebook link-click metrics (Panels C and D in Table 4). We find that the Treatment 1 ad click-through rate is higher, by a statistically significant amount, than those for the Control and Treatment 2 Table 4. Tests of significance of the difference between messages, April 22–25, 2018, experiment ads (p = 0.0328, p = 0.0428). We also find that when using sessions and new users to calculate click-through rates, the Control and Treatment 1 ads have statistically significant higher rates than the Treatment 2 ad (p < 0.01).

Regarding engagement, we show the tests on the difference in post-reaction, post-share and page likes rates between the ads in Table 4, Panels E, F and G. Only for the page likes rate do we observe that the Treatment 1 ad has a statistically significantly higher rate than the Control and Treatment 2 ads(p < 0.01).

Discussion

Based on the analysis of the data collected in our two real-world Facebook experiments, we find that the ad that used an injunctive norm emphasizing peer effect (Treatment 1) was the most successful at getting Hispanic women interested in learning more about retirement planning. When using the click-through rates calculated using Facebook data, the Treatment 1 ad was more successful than the Treatment 2 ad in all instances and more successful than the Control ad in most cases (two out of four cases). When using the click-through rate calculated with the number of sessions and new users from Google Analytics, we found that the Treatment 1 ad was more successful than the Control and Treatment 2 ads in all cases. We prefer this click-through rate to the one calculated using link-clicks from Facebook, as we discussed previously.

We also estimated whether the difference in the rates was significant with a statistical power of 80% and found that, for all click-through rates in the July 2016 experiment, the difference between the Treatment 1 and Control ads was significant. According to power calculations, the difference between the Treatment 1 and Treatment 2 ads was also significant for all click-through rates for the July 2016 experiment.

For the April 2018 experiment, power calculations indicated that the difference between the Treatment 1 and Control ads was not statistically significant for the four versions of the click-through rates. Our budget was much smaller for the second experiment, which led to a smaller sample. If we calculate the statistical power for the difference in the four versions of the click-through rates between the Treatment 1 and Control ads in the April 2018 experiment using the sample size from the July 2016 experiment, we find that the difference is significant. While this is a hypothetical calculation, it allows us to see that if the budget would have been higher, we might have reached similar results to what we observed in the July 2018 experiment.

Using the Facebook Ads Manager platform, we obtained disaggregated impressions, link-clicks and link-click rates by age, state and impression device. Disaggregated data are shown in Tables 5 and 6 for the July 2016 and April 2018 experiments, respectively. In relation to age, most impressions were shown to participants aged 45–55 years (65% and 64%). We also found that our ads were more successful among women aged 45–55 years since click-through rates (using link-clicks, version 1) were much higher than for the other age group (1.72 versus 2.56; 2.03 versus 3.28). The fact that older participants were exposed to more impressions as a result of Facebook’s optimizing strategy and that they showed higher click-through rates is expected, since this group may be more interested in retirement, given that retirement is closer and more salient to them.

Table 5.

Impressions and link-clicks disaggregated by age, state and device for the three messages, July 21–25, 2016, experiment

	Impressions	Impressions (%)	Link-clicks	Link-clicks (%)	Link-click rate
By age
33–44	63,463	35.30	1094	26.88	1.72
45–55	116,343	64.70	2976	73.12	2.56
By state
Arizona	12,336	6.86	284	6.98	2.30
California	95,368	53.04	2187	53.73	2.29
New Mexico	4074	2.27	74	1.82	1.82
Texas	68,029	37.83	1519	37.32	2.23
Other	–	0.00	6	0.15	–
By device
Android smartphone	138,265	76.90	3054	75.04	2.21
Android tablet	5113	2.84	118	2.90	2.31
Desktop	647	0.36	17	0.42	2.63
iPad	4598	2.56	126	3.10	2.74
iPhone	30,396	16.90	741	18.21	2.44
iPod	160	0.09	1	0.02	0.63
Other	628	0.35	13	0.32	2.07

Notes: Impressions and link-clicks disaggregated by age, state and device from Facebook reports. The ‘Other’ category in the state subgroup includes places in Mexico (Baja California, Distrito Federal, Jalisco, Michoacán, Sonora and Tamaulipas). For the ‘Other’ category in the state subgroup, Facebook did not provide numbers of impressions.

Table 6.

Impressions and link-clicks disaggregated by age, state and device for the three messages, April 22–25, 2018, experiment

	Impressions	Impressions (%)	Link-clicks	Link-clicks (%)	Link-click rate
By age
33–44	40,168	36.26	816	26.05	2.03
45–55	70,603	63.74	2316	73.95	3.28
By state
Arizona	7882	7.12	239	7.63	3.03
California	54,037	48.78	1627	51.95	3.01
New Mexico	2782	2.51	77	2.46	2.77
Texas	46,070	41.59	1189	37.96	2.58
By device
Android smartphone	72,812	65.73	2030	64.81	2.79
Android tablet	748	0.68	20	0.64	2.67
Desktop	113	0.10	1	0.03	0.88
iPad	1636	1.48	47	1.50	2.87
iPhone	35,330	31.89	1032	32.95	2.92
iPod	31	0.03	0	0.00	0.00
Other	101	0.09	2	0.06	1.98

Notes: Impressions and link-clicks disaggregated by age, state and device from Facebook reports. The ‘Other’ category in the state subgroup is not included here since there were no other states/regions in this experiment.

In relation to the state, we observe that individuals in California and Texas were shown the most impressions relative to individuals in other states in both experiments. Click-through rates using link-clicks were higher among study participants in Arizona and California in both experiments. Finally, we note that most individuals in our study were reached through either an Android smartphone in both experiments (77% and 66% of impressions) or an iPhone (17% and 32% of impressions). Very few people in our study were reached through a desktop in both experiments (less than 1% of impressions).⁹

It is important to note that when we first ran our experiment on Facebook in July 2016, we were able to run the experiment right away. For our second experiment conducted in April 2018, because we were targeting a specific minority group, our ads had to be reviewed and approved by Facebook in order to comply with Facebook’s non-discrimination policies. While we make comparisons between our two experiments, it is important to note that the environment in Facebook was very different in 2016 than it was in 2018. A limitation of our study that is worth mentioning is that our targeting strategy relies entirely on the information that individuals provide in Facebook about their race/ethnicity, language and state of residence, which might not be 100% accurate.

Another limitation of our study is that we could express the distinction between peer effects and familial values in several ways. The message used for Treatment 1 is likely to be more successful than the message for Treatment 2 because an ad worded like Treatment 1 readily evokes self-presentation, status and appearance (the peer injunctive). On the other hand, in the message used for Treatment 2, the automatic connection may not hold when the feeling is about one’s family instead of oneself. We only tested two messages, and further analysis that exploits different messages could be beneficial in order to determine whether our findings are consistent across different messages.

Conclusion

Our study provides a couple of key insights for policy-makers, financial institutions and researchers interested in promoting financial planning for retirement among Hispanic women. When reaching out to the population of interest with the purpose of promoting retirement planning and saving, messages based on injunctive norms that are centered on peer influence and self-empowerment may be more successful than messages based on injunctive norms centered on the family. This was a surprising finding, as we initially hypothesized that inherent cultural values might have a stronger influence among different racial and ethnic groups than pure peer effects. We were expecting that a message centered on the family would be more effective when talking about retirement planning, given the large literature on the importance of family networks among Hispanics of older ages, as discussed previously.

Based on our experience in running these experiments, we found Facebook to be a useful social media tool for delivering information about retirement planning and saving to Hispanic women with certain demographic characteristics. Moreover, experiments like this allow us to further understand our audience and learn more about their behavior on Facebook. Disaggregating our link-clicks by age, state and device provides us with useful information for the design of future interventions that aim to promote retirement saving among Hispanic women. In our study, we find that our ad campaigns were more successful among women aged 44–55 years (younger boomers). While we would like to get Hispanic women to save for retirement as early as possible, knowing that there is more interest in retirement planning among younger boomers will be important in the design of a program that targets women of that specific age. Getting women to start saving for retirement in their 40s will allow them to build wealth for approximately 20–30 years – a significant amount of time.

In addition, the results showed that our campaign had the most number of impressions among individuals in California and Texas. We hypothesize that these individuals might possess a greater awareness of retirement saving issues. For example, in the state of California, such awareness may be a result of the California Secure Choice Retirement Savings Act, which was signed into law in September 2016 to increase participation in retirement saving plans, particularly among workers at small businesses without employer-sponsored plans.¹⁰ Our study was conducted in July 2016, overlapping with the discussion of this law by the media and policy-makers. Further research on how California and Texas might be different from the other states in relation to interest in retirement saving might be warranted.

Another valuable finding about our population of interest was that the vast majority of individuals were reached through a smartphone – specifically, an Android smartphone – and not a desktop computer in this study. This finding is important as it provides evidence that the population is harder to reach through a desktop channel than through a mobile device on Facebook, which is something that should be considered when designing future interventions to promote financial planning for retirement among Hispanic women. An intervention that leverages this technology might be a simple and cost-effective way to promote better financial management practices among Hispanic women.

This study could inform interventions that aim at promoting retirement saving among Hispanics. For example, when researchers or policy-makers (or both) design an educational intervention to promote retirement planning and saving, they could use a message centered on what we found to be most successful in this study.¹¹ Our study can provide some insights into future interventions that focus on behavioral change. For example, researchers could conduct split testing in Facebook following our methodology to find out what messages will be more successful in promoting behavioral change.