Testing the keepin’ it REAL Substance Use Prevention Curriculum among Early Adolescents in Guatemala City.

Abstract

This article describes a test in Guatemala City of Mantente REAL, a linguistically adapted version of the keepin’ it REAL universal substance use prevention curriculum for early adolescents that teaches culturally grounded drug resistance, risk assessment and decision making skills. Academic researchers collaborated with a local non-profit to recruit and randomize 12 elementary schools in Guatemala City to intervention and comparison conditions. Regular classroom teachers were trained to deliver the ten-lesson Mantente REAL (MR) manualized curriculum to sixth grade students. Parents provided passive consent and students gave active assent for data collection, which occurred between February 2013 and September 2014. Two academic year cohorts of students participated (n=676; 53% male; M age=12.2). All students completed a pretest questionnaire before the curriculum lessons began in intervention schools and a posttest (87% matched) four months later, one month after the final lesson. We assessed the MR intervention with paired t tests, effect sizes (Cohen’s d), and general linear models adjusted for baseline, attrition, non-linear distributions, and school-level clustering. Results indicated that MR can be an effective school-based prevention approach in Guatemala. The MR participants reported pretest-to-posttest changes in desirable directions on substance use behaviors, attitudinal antecedents of substance use, and acquisition of drug resistance skills. The comparison group generally changed in undesirable directions. In linear models the MR participants, relative to the comparison group, reported less cigarette and marijuana use, less positive drug use expectancies, and greater use of drug resistance skills. Intervention effect sizes were between .2 and .3.

The Context for Youth Substance Use Prevention in Guatemala

The harmful effects of substance misuse and abuse constitute a major public health threat globally, accounting for many preventable deaths, diseases and injuries (WHO, 2014). While evidence-based substance use prevention programs for adolescents have been developed and are increasingly implemented in high income countries, these programs are only beginning to be adapted and tested in middle and low income countries (Catalano et al., 2012). The current study tested a linguistically adapted youth substance abuse prevention program from the USA, keepin’ it REAL/Mantente REAL, that was implemented in high-risk neighborhoods of Guatemala City, and delivered by trained classroom teachers to students in their last year of elementary school. The study tested the efficacy of the intervention, relative to a no-intervention comparison group, across an array of substance use related outcomes, and estimated effect sizes of the intervention.

Guatemala exemplifies social, economic and political dislocations found in some low income countries that combine to increase the vulnerability of youth to substance misuse and exacerbate the need for effective prevention interventions: rapid population growth and increasing urbanization, political and community violence, and accessibility of illicit drugs. Guatemala has the highest fertility and population growth rates in Latin America and the youngest population (United Nations, 2017). About half of its nearly 17 million residents are younger than 19 years of age (USAID, 2016a) and just under half are urban residents (Pan American Health Organization, 2017). Guatemala’s recent sociopolitical history was dominated by a long and bloody civil war that killed over 200,000 people, orphaned over 100,000 children, and displaced an estimated one million residents (Hernández-Bonilla, 2017). The resulting widespread disintegration of family units negatively affected youth socialization and reduced access to education as families prioritized more basic needs and infrastructure and funding for public education declined (Chamarbagwala & Morán, 2011; Kurtenbach, 2014). Further, behavioral health problems emerged after many Guatemalans were exposed to extreme violence during the war, with spikes in alcohol-related and post-traumatic stress disorders (Puac-Polanco et al., 2015). The war’s end also brought an increase in alcohol and other drug consumption after peace accords led to the opening of borders and more drug trafficking (McIlwaine & Moser, 2004). The lingering effects of civil war continue to adversely affect youth development (Sabin, Cardozo, Nackerud, Kaiser, & Varese, 2003). Family disruption, a lack of social cohesion, and limited educational and employment opportunities are directly linked to increases in gang activity, community violence, drug trafficking, and exposure to illicit substance use (Rodríguez & García Santiago, 2007). Concentrated drug trafficking and violence in Guatemala’s cities (United Nations Office on Drug and Crime, 2013) help explain why youth who reside in urban areas and living in extreme poverty are at greater risk of engaging in substance abuse than their rural counterparts (McIlwaine & Moser, 2004). For Guatemalan youth, seeking stress relief from violence exposure is a significant risk factor for substance abuse (Kliewer & Murelle, 2007).

Other social, cultural and political factors increasing the vulnerability of Guatemalan youth to substance use include relatively tolerant attitudes toward substance use, lack of policies targeting substance abuse, and limited policy enforcement. Permissive attitudes toward substance use in Guatemala may stem from cultural norms that approve of celebratory drinking (Kanteres, Lachenmeier, & Rehm, 2009), as well as beliefs that drug use facilitates social interactions and provides new pleasurable experiences (Fortin & Bertrand, 2013). Furthermore, the Guatemalan government does not have a strong record of enacting or enforcing policies to combat substance abuse, or of implementing substance use prevention programs (Sebrié et al., 2012).

The factors described above create high-risk conditions exposing Guatemala’s urban youth to alcohol, tobacco, other drugs, and violence from an early age. Without adequate prevention and early intervention efforts, the youth are at elevated risk of engaging in substance use. Since the civil war, the very limited and scattered prevention efforts for Guatemalan youth have focused on violence prevention, especially demonstration projects targeting high risk youth (RTI International, 2015). The national plan for combatting addiction and drug trafficking in Guatemala recommended the implementation of prevention programs in schools and the training of school teachers (SECCATID, 2009) but evidence-based substance use prevention programs for the general population of Guatemalan youth are not yet institutionalized in schools.

Available data on youth substance use in Guatemala is limited, in part because school-based surveys fail to represent large segments of the youth population that drop out after primary school. However, existing studies suggest that substance use is less prevalent among Guatemalan youth than in the USA but initiated at relatively early ages. Alcohol and tobacco are the substances used most frequently by Guatemalan youth between 12 and 18 years old (Díaz, Delva, & Anthony, 1998), with lifetime prevalence rates of 26% for alcohol and 22% for cigarettes (Dormitzer et al., 2004). Some research suggests that alcohol use by adolescent boys in Guatemala begins around 12 to 14 years of age (McIlwaine & Moser, 2004). Although they are much less involved with illicit drugs, Guatemalan youth who do use them initiate use at alarmingly young ages. A national survey of Guatemalan students between 11 and 22 years old reported lifetime prevalence of 9.4% for inhalants, 7.5% for marijuana, and 2.5% for cocaine (Bolívar Díaz, n.d., as cited in Urizar, 2014). However, among users, the mean age of first use was 5 years for inhalants, 6 years for marijuana and cocaine among males, and 8 years old for cocaine among females. This survey also found evidence of trends toward increasing rates of substance abuse over the last 20 years. In assessing the prevalence, amounts and severity of alcohol consumption cross-nationally, the World Health Organization rates Guatemala as 4 out of 5, with 5 representing the gravest level of risk to population health (WHO, 2014).

Prevention programs can reduce substance abuse among youth by addressing risk factors such as those described above while also increasing protective factors. The most effective school-based universal prevention programs teach drug-resistance and life-skills in interactive formats that allow youth to apply and rehearse the skills, in contrast to programs that only relay information or employ scare tactics (Tobler et al., 2000). Programs can help youth understand social factors (e.g., peer expectations) that lead to risky behaviors while fostering a variety of adaptive skills including problem-solving, decision making, risk assessment, cognitive and behavioral coping strategies, and interpersonal skills for refusing substance use (Botvin, 1990).

Mantente REAL

Mantente REAL is a Spanish language translation of the keepin’ it REAL (kiR) program, a manualized universal prevention intervention originally developed and tested with a predominantly Latino/a youth sample in the U.S. (Marsiglia & Hecht, 2005). The kiR program is grounded in a conceptual model that integrates ecological risk and resiliency theory, communication competence theory, and narrative theory (Gosin, Marsiglia, & Hecht, 2003). The program is designed to delay or reduce substance use among early adolescents by increasing youth’s repertoire of culturally congruent drug resistance skills and promoting non-permissive substance use norms and attitudes. Specifically, kiR consists of 10 weekly lessons delivered in classrooms by the students’ regular school teachers. In the sessions students build a repertoire of successful resistance strategies and learn the communication and risk assessment skills to apply them in different contexts and with different people, which helps them navigate drug-related risky situations more safely (Marsiglia & Hecht, 2005; Wright, Nichols, Graber, Brooks-Gunn & Botvin, 2004). The acronym REAL – Refuse, Explain, Avoid, Leave – represents the four drug resistance strategies used most commonly by youth: refuse substance offers with a direct no, explain why you decline, avoid substance offer situations, or leave them.

The U.S. Substance Abuse and Mental Health Services Administration recognized the kiR curriculum as a “Model Program” based on the evidence of its efficacy in reducing youth substance use in diverse USA samples (Hecht et al., 2003; Kulis et al., 2005; Marsiglia, Kulis, Yabiku, Nieri, & Coleman, 2011). Randomized controlled trials showed that kiR delays the onset of adolescent substance use (Hecht et al., 2003), and was also effective in reducing substance use among youth who had already initiated use (Kulis, Nieri, Yabiku, Stromwall, & Marsiglia, 2007). Evidence of effectiveness with Latino heritage youth is strong. Among Mexican American youth in the southwestern USA, kiR reduced alcohol, tobacco, and marijuana use, while increasing anti-drug attitudes and norms (Kulis et al., 2005). Spanish language versions of kiR have been tested and shown to be effective in Mexico (Marsiglia et al., 2014; Marsiglia, Kulis, Booth, Nuño-Gutierrez, & Robbins, 2015) and in Uruguay (Marsiglia et al., 2017).

The present study examines the efficacy of Mantente REAL (Rechaza, Explica, Alejate, Levantate), a linguistically adapted Spanish-language version of keepin’ it REAL, in Guatemala City. The study aim was to test whether the Mantente REAL program prevented substance use among a community sample of Guatemalan early adolescents, preserved non-permissive attitudes toward youth substance use, and imparted effective drug-resistance skills.

Methods

Impetus for the Study

A non-profit organization in Guatemala, U Yum Cap, partnered with academic researchers from the USA to introduce and assess effective school-based substance use prevention programs in Guatemala. The local research team from the non-profit organization selected the Mantente REAL prevention curriculum as a promising intervention and asked the university researchers who created that curriculum to collaborate on an implementation and trial of Mantente REAL in Guatemala. For this initial trial the joint research teams decided to implement the most widely disseminated and evidence-backed version of kiR (Marsiglia & Hecht, 2005) after a translation into Spanish that was verified through a review by local teachers.

Setting, School Selection, and Participants

The study site, Guatemala City, is the nation’s capital and largest city, with 3.3 million residents, making it also the largest city in Central America. Spurred by migration from rural areas, political instability, and relative lack of economic opportunities elsewhere, the metropolitan population has exploded in recent decades, causing severe strains on infrastructure for transportation, sanitation, utilities, and education (USAID, 2016a; Valladares Cerrezo, 2003; Worldmark Encyclopedia of the Nations, 2007). The Guatemala public education system includes compulsory primary education (elementary school, grades 1 through 6) for children ages 7–12 years old, followed by three years of lower secondary education (middle school) that is legally compulsory but lacks enforcement and resources; thus, many students do not enter or complete middle school (EPDC, 2012; UNESCO, 2010). Although Guatemala has achieved nearly universal enrollment in initial primary education and dramatic increases in 1^st grade completion rates, only three-fourths of those in primary school graduate from 6^th grade, and less than 40% enroll in lower secondary education (middle school) (USAID, 2016b). The steep decline in enrollment as Guatemalan youth transition from primary to lower secondary school is a decisive reason to target the end of primary school as an opportune time to implement and test a universal prevention curriculum like Mantente REAL.

The 676 study participants in this study were enrolled students in the 6th grade in 12 elementary schools (primarias) in Guatemala City (See CONSORT diagram, Figure 1). Schools were selected based on their location in four neighborhoods where violent crime rates are relatively high. The local research team visited 21 primary schools located in the targeted zones and made presentations about the project to the school principals. Of these schools, 12 agreed to participate. The schools were then randomized into intervention conditions, with seven schools assigned to receive the Mantente REAL prevention curriculum and five schools serving as a no intervention comparison group. Two cohorts of students participated at each school, starting at the beginning of the 2013 or the 2014 academic year. The number of participating students per school ranged from 31 to 105, and class sizes varied from 14 to 40 students. All enrolled sixth grade students in the study schools were eligible to be included as participants.

Figure 1.

CONSORT Diagram for Guatemala Mantenta REAL Efficacy Trial

Survey Administration and Human Subjects Protections

With the approval of the researchers’ university IRB and in accordance with school policies, parents provided passive consent and students gave active assent to complete questionnaires. The schools notified parents that their child’s school would be taking part in a research study, providing information and a contact number for any questions, concerns, or to exclude their child from the survey data collection. Survey proctors from the non-profit organization informed students that the questionnaires were part of a university research project, their participation was voluntary, and answers would remain confidential. Students signed an assent to participate. The research team received no reports of parents denying consent, and no students declined to provide assent to the survey data collection.

Before delivering the curriculum lessons in the intervention schools, students in all study schools completed a one-hour self-administered pretest questionnaire. Four months later, and about one month after the intervention schools completed delivery of Mantente REAL, all students completed a posttest questionnaire. Because the program implementation and survey data collection were conducted as part of regular school activities, nearly all enrolled students were present at both the pretest and posttest. If five or fewer students were absent on a scheduled survey collection day the absentees had an opportunity to take the survey later; if more than five students were absent, survey collection was rescheduled for the class. The non-profit organization conducted data collection and data entry and sent the university researchers a data set without any individually identifying information. We matched student pretests to posttests with a confidential unique identifier that students created for themselves, using combinations of numbers and letters. Using this method attrition was low: 87% of the pretests had a matched posttest. Although some of the minimal loss to follow-up was due to students transferring to another school, most of the attrition was due to failure to match pretests to posttests because of missing or inconsistent information in the unique identifier that students created.

Curriculum Training and Delivery

All 6^th grade teachers in intervention schools received training over two days to deliver the manualized Mantente REAL curriculum. Training was led by the local project director, who, in turn, had received intensive training-of-trainers instruction from the original keepin´ it REAL curriculum trainers on the university research team. Each implementing teacher received a teacher curriculum manual and an accompanying student handbook for every student. The local team evaluated the schools’ implementation needs and provided any needed support for equipment and materials, such as laptop computers, DVD players, and visual aids. The students’ regular 6^th grade teachers delivered the Mantente REAL lessons during regular school hours over about a three-month period, ranging from 11–14 weeks, with some lessons spanning multiple classes in a week. Local team members observed several lessons to monitor teacher fidelity to the curriculum manual. These were informal assessments, for the purpose of providing feedback to the teacher about the curriculum’s intended content, activities, and format. After observations, the local research team met informally with teachers and principals to discuss their progress, obstacles, concerns, and potential solutions.

Participant Characteristics

Table 1 presents a demographic profile of the respondents. Their gender composition was somewhat skewed, with more males (54%) than females (47%). Students were age typical for the 6^th grade, with 90% between the ages of 11 and 13. Most students (71%) had been born in Guatemala City, and 83% had lived in the capital for 10 years or more. A large plurality of the students lived with both parents (80%), and in large households, with about seven persons on average. Students were generally from lower socioeconomic status families as indicated by the educational level of parents. Almost half (45%) of the parents had only a primary school education or less, and two-thirds (67%) did not complete high school. Student grades, on a scale corresponding to a 1-through-4 grade point average, were centered around a high C or B-.

Table 1.

Demographic Characteristics of Participants, by Intervention Condition

	Mantente REAL (n=448)	Comparison Group (n=228)	Total (N=676)	Difference Test: Mantente REAL versus Comparison Group
Gender
Female	45.4%	48.7%	46.6%	Χ2 = 0.6, 1 df, p=.44
Male	54.4%	51.3%	53.4%
Living Arrangement
Both Parents	79.6%	80.6%	79.9%	Χ2 = 0.4, 2 df, p=.83
Single Parent	17.8%	17.6%	17.7%
Other Relative	2.6%	1.9	2.4%
Birthplace
Guatemala City	72.9%	67.8%	71.2%	Χ2 = 2.6, 3 df, p=.45
Nearby City	16.1%	20.3%	17.5%
Elsewhere in Guatemala	7.5%	7.0%	7.3%
Other Country	3.6%	4.8%	4.0%
Residence in Guatemala City
Less than 3 Year	4.5%	4.8%	4.6%	Χ2 = 0.8, 2 df, p=.65
3–10 Years	105%	12.9%	12.1%
> 10 Years	82.6%	84.6%	83.3%
Highest Parental Education
Less than Primary (Primaria)	16.1%	24.6%	19.0%	Χ2 = 28.2, 4 df, p=.000
Primary School	21.9%	33.3%	25.8%
Middle School (Secundaria)	24.2%	20.6%	23.0%
High School (Bachillerato)	18.3%	7.9%	14.8%
Beyond High School	19.5%	13.6%	17.5%
Age (Mean)	(12.19)	(12.31)	(12.23)	t = 1.49, p=.16
Household Size (Mean)	(6.70)	(7.21)	(6.87)	t = 1.769, p=.08
Average School Grades (Mean)	(2.80)	(2.64)	(2.76)	t = 2.16, p=.03

Tests of baseline differences between the Mantente REAL and comparison groups appear in the last column of Table 1. There were no differences between these groups in gender composition, age, household size, parental presence in the home, place of birth, or length of residence in Guatemala City. However, the Mantente REAL group reported somewhat higher levels of parental education and higher average school grades. Accordingly, in tests of intervention effects we investigated whether the effects persisted after controlling for these baseline differences in grades and parental education; the effects were essentially unchanged and these controls were dropped from the final models.

Outcome Measures

Study outcomes were validated measures of substance use behaviors, an array of antecedents of substance use, the drug resistance skills targeted in the prevention curricula, other risk behaviors, and self-esteem. Table 2 details the source of the measures, question wording, and response options, and compares scale reliability in the study sample with the original source. The pretest and posttest included identical items measuring all outcomes. Three measures gauged the amount of recent (last 30 day) alcohol, cigarette, and marijuana use, using developmentally appropriate questions for this age group (Hansen & Graham, 1991). We examined key antecedents of youth substance use initiation, each measured as the scale mean of several component items: substance use intentions, permissive drug norms, and positive substance use expectancies (Hecht et al., 2003). These antecedents provide valuable assessments of the effectiveness of school-based prevention programs for the majority of early adolescents who have not yet initiated substance use but are entering a developmental period when experimentation begins. For the measures of substance use antecedents, higher values indicate stronger pro-drug orientations. We examined the students’ use of the different REAL strategies (refuse, explain, avoid, leave) to deal with offers of alcohol by combining questions in three ways. The first calculated the mean frequency that students used the strategies. The second counted the number of different strategies they used, i.e., the size of their REAL “repertoire.” The third calculated a mean for the likelihood that they would use each of the strategies in a hypothetical situation, if a friend offered them a beer at a party (Hecht et al., 2003). We also utilized a scale measuring susceptibility to negative peer influence, such as succumbing to pressure to use substances, skip school, or vandalize (Luengo, Romero, Gómez-Fraguela, Guerra, & Lence, 1999). Three additional individual items recorded the frequency that the student engaged in robbery, hitting someone, or a school fight (CDC, 2015). The Mantente REAL intervention might decrease vulnerability to these types of antisocial behavior by increasing positive decision making and resistance skills. The final measure was a scale composed of negative self-esteem items, e.g., feeling you have little to be proud of (Rosenberg, 1965).

Table 2:

Outcome Measures

Outcome	Questions	Responses and Scoring	Scale Reliability in Sample	Reference & Published Reliability
Alcohol Amount	(1 question) - In the last 30 days, how many drinks of alcohol have you had?	0 to 5: none, 1 or part of 1, 2–3, 4–7, 8–15, over 15		Hansen & Graham (1991)
Cigarette Amount	(1 question) - In the last 30 days, how many cigarettes have you smoked?	0 to 5: none, a puff, 1 or part of 1, 2–3, 4–5, > 5		Hansen & Graham (1991)
Marijuana Amount	(1 question) - In the last 30 days, how many marijuana cigarettes have you smoked?	0 to 5: none, a puff, 1 or part of 1, 2–3, 4–5, > 5		Hansen & Graham (1991)
Substance Use Intentions	(3 questions) - If you had the chance this weekend, would you use: alcohol? …cigarettes? …marijuana?	1 to 4: definitely no, no, yes, definitely yes	α = .94	Hecht et al. (2003) α = .82
Permissive Drug Use Norms	(3 questions) - Is it okay for someone your age to: drink alcohol? …smoke cigarettes? …use marijuana?	1 to 4: definitely not OK, OK, not OK, definitely OK	α = .97	Hecht et al. (2003) α = .86
Positive Substance Use Expectancies	(6 questions) - Do you agree or disagree? Drinking alcohol makes parties more fun. Smoking cigarettes helps people relax. Smoking marijuana makes it easier to be part of a group. Using drugs every now and then helps people deal with problems. You have to try drugs to be able to talk to others about their effects. Using drugs would give me new experiences and make me happy.	1 to 4: strongly disagree, disagree, agree, strongly agree	α = .96	Hecht et al. (2003) α = .78
REAL Drug Resistance Strategies: Frequency and Repertoire	(4 questions) – When you were offered alcohol in the last 12 months how often have you … Said ‘No’ without saying why [Refuse]; …Said ‘No’ and gave a reason why [Explain]; …Left the situation or the place [Leave]. How often in the last 12 months have you avoided situations or places where you thought you might be offered alcohol [Avoid]	Frequency 0 to 4: never, once, 2–3, 4–5, >5 times. Mean. Repertoire Count (0 to 4) of items with responses of once or more often.	α = .72	Hecht et al. (2003); α not reported
REAL Drug Resistance Strategies: Hypothetical	(3 questions) – If a friend offered you a beer at a party, would you… Say ‘No’ without giving a reason why [Refuse]; …Give an explanation or excuse for not drinking the beer [Explain]; …Leave without drinking the beer [Leave].	1 to 4: Definitely no, probably no, probably yes, definitely yes	α = .77	Hecht et al. (2003); α not reported
Susceptibility to Negative Peer Influence	(6 questions) – I would… smoke a cigarette if a friend dared me to even if I didn’t want to. …go along with my best friend and skip school. …feel bad if I didn’t drink at a party where everyone was drinking. …go to the movies with friends even if I had to study for an exam. …rip pages from library books if my friends dared me to do it. I hang out with people who always get into trouble.	Yes or no questions: Scale is proportion (0 to 1.0) of “yes” responses	α = .66	Luengo et al. (1999) α = .57
Robbery Hit in a Fight School Fight	(3 individual questions) – In the last 12 months how often have you …robbed someone? … been hit in a fight? … participated in a fight at school?	0 to 4: never, once, twice, three times, 4 or more times		CDC (2015)
Poor Self Esteem	(2 questions) – In the last week have you felt this way about yourself?: I felt I did not have much to be proud of. At times I felt I am not good at all.	1 to 4: strongly disagree, disagree, agree, strongly agree	α = .56	Rosenberg (1965) α = .72 to .88

Analysis Strategy

We verified the psychometric properties of the measures with reliability and factor analyses and made adjustments to one measure, susceptibility to negative peer influences, by dropping two items that did not refer directly to peer influence. To assess the intervention outcomes, we examined changes from pretest to posttest with pairwise t tests, within and between intervention conditions (Mantente REAL or comparison group), to show the direction and statistical significance of these changes. Then, general linear models tested for intervention effects using dummy variable contrasts of Mantente REAL versus the comparison group, controlling for the outcome as measured at the baseline pretest. Finally, we estimated the intervention effect size using Cohen’s d, comparing the size of mean changes in outcomes in the intervention group to that of the comparison group. All tests employed full-information maximum likelihood (FIML) estimation in Mplus 7.0 (Muthén & Muthén, 2012) to account for attrition to the posttest (13%) and item missing data, using a robust maximum likelihood estimator (MLR) to adjust for non-normal distributions, and adjusting for school-level clustering.

Results

Table 3 presents means, standard deviations, and mean changes in the outcomes from pretest to posttest, separately by intervention condition. In the Mantente REAL group all the changes in outcomes were in the desirable direction, toward lower levels of substance use, weaker endorsement of pro-drug attitudes, more use of drug resistance strategies, less susceptibility to negative peer influences, less frequent antisocial behavior, and less negative self-esteem. Most of these changes attained (p ≤ .05) or approached (p ≤ .10) statistical significance: the reductions in alcohol and cigarette use, adherence to pro-drug norms, negative peer susceptibility, and poor self-esteem, as well as increases on all the measures of use of the REAL strategies. In the comparison group, the changes were generally in an undesirable direction, toward more substance use, anti-social behavior, and susceptibility to negative peer influence, and toward less use of the drug resistance strategies. The comparison group reported significant increases in cigarette and marijuana use, positive drug expectancies, and robbery. However, the comparison group also reported significant decreases on two substance use antecedents: intentions to use and pro-drug norms. The last columns compare changes in the Mantente REAL group relative to changes in the comparison group. Ten of fourteen outcomes changed in a statistically significant more desirable direction in the Mantente REAL group than in the comparison group: all measures of recent substance use, anti-social behaviors, drug use expectancies, self-esteem, and actual use of the REAL strategies.

Table 3.

Pretest to Posttest Changes in Outcomes, by Intervention Condition

	Mantente REAL (n = 448)						Comparison Group (n=228)
	Pretest		Posttest		∆ Posttest – Pretest		Pretest		Posttest		∆ Posttest – Pretest		∆ MR versus ∆ Comparisona
	M	SD	M	SD	Diff.	t-test	M	SD	M	SD	Diff.	t-test	Diff.	t-test
Alcohol Amount	0.149	0.542	0.100	0.420	−0.049	−1.855†	0.091	0.455	0.128	0.513	0.037	0.888	−0.085	−1.743†
Cigarette Amount	0.121	0.590	0.064	0.491	−0.057	−1.907†	0.049	0.288	0.216	0.867	0.167	2.648**	−0.225	−3.211**
Marijuana Amount	0.032	0.263	0.014	0.138	−0.018	−1.614	0.022	0.486	0.071	0.535	0.067	2.154*	−0.049	−1.679†
Intentions to Use	1.334	0.532	1.288	0.498	−0.046	−1.513	1.462	0.555	1.337	0.559	−0.124	−2.511*	0.079	1.361
Pro-drug Norms	1.344	0.502	1.253	0.455	−0.091	−3.269**	1.388	0.507	1.253	0.429	−0.135	−3.384**	0.044	0.907
Positive Drug Expectancies	1.885	1.038	1.812	1.025	−0.072	−1.212	1.793	1.013	2.034	1.152	0.242	2.501*	−0.314	−2.765**
REAL Frequency	0.487	0.774	0.603	0.855	0.117	2.531***	0.493	0.836	0.466	0.730	−0.027	−0.426	0.144	1.834†
REAL Repertoire	0.887	1.223	1.195	1.439	0.307	4.398*	0.925	1.245	0.910	1.203	−0.015	−0.154	0.323	2.655**
REAL Hypothetical	2.524	1.065	2.719	1.130	0.196	2.964**	2.530	1.094	2.598	1.145	0.068	0.719	0.128	1.114
Neg. Peer Susceptibility	0.065	0.145	0.053	0.138	−0.012	−1.651†	0.052	0.110	0.059	0.161	0.007	0.597	−0.019	−1.392
Robbed Someone	0.137	0.571	0.105	0.495	−0.032	−0.915	0.079	0.332	0.156	0.569	0.077	1.723†	−0.109	−1.918†
Hit in a Fight	0.306	0.716	0.248	0.675	−0.059	−1.530	0.229	0.704	0.294	0.823	0.066	1.229	−0.125	−1.891†
Fought at School	0.304	0.786	0.262	0.716	−0.042	−1.110	0.215	0.639	0.295	0.818	0.080	1.614	−0.122	−1.957†
Poor Self-Esteem	2.987	1.258	2.755	1.304	−0.232	−3.445**	2.976	1.197	3.009	1.357	0.033	0.332	−0.265	−2.226*

^aPretest to posttest change in Mantente REAL minus pretest to posttest change in comparison group.

^†p < .10.

^*p < .05.

^**p < .10.

^***p < .001

Table 4 summarizes the direct tests of differences in outcomes between the Mantente REAL and comparison groups using baseline-adjusted general linear models. These models predict the outcome as measured at the posttest while controlling for the pretest report on the same outcome, which makes the models a test of changes in outcomes from pretest to posttest. The dummy variable contrast of the intervention conditions shows that changes were relatively more favorable in the Mantente REAL group on all outcomes except adherence to pro-drug norms. Five of these were statistically significant. The Mantente REAL students reported relatively less use of cigarettes and marijuana than the comparison group, less endorsement of positive drug expectancies, more frequent use of the REAL strategies and a larger REAL repertoire, and less negative self-esteem. As indicated in the last column of the table, effect sizes for Mantente REAL were strongest for cigarette use, approaching a moderate size effect of .3, with smaller effects around .2 for marijuana use, positive drug use expectancies, the REAL repertoire and poor self-esteem. Effect sizes for many of the remaining, non-significant intervention effects were smaller, around .15, suggesting that a considerably larger sample would be required to demonstrate the efficacy of the intervention in addressing these outcomes.

Table 4.

Intervention Effects

	Intercept		Outcome At Pretest		Mantente REAL versus Comparison		R- Squar e	N	Effect Size b d
	Est.a	SE	Est.	SE	Est.	SE
Alcohol Amount	0.182*	0.083	0.393***	0.107	−0.031	0.047	0.154	651	.150
Cigarette Amount	0.294*	0.117	0.334**	0.122	−0.129*	0.058	0.122	638	.296
Marijuana Amount	0.200*	0.094	0.503***	0.112	−0.097*	0.048	0.261	649	.211
Intentions to Use	1.856***	0.176	0.269***	0.053	−0.019	0.051	0.071	671	.115
Pro-drug Norms	2.023***	0.197	0.287***	0.049	0.010	0.044	0.082	674	.074
Drug Expectancies	1.425***	0.112	0.266***	0.045	−0.104**	0.038	0.080	676	.235
REAL Frequency	0.344***	0.041	0.355***	0.034	0.081***	0.019	0.132	669	.149
REAL Repertoire	0.341***	0.043	0.419***	0.038	0.106***	0.017	0.186	676	.217
REAL Hypothetical	1.632***	0.107	0.274***	0.043	0.053	0.033	0.077	654	.091
Peer Susceptibility	0.229*	0.094	0.424***	0.075	−0.033	0.062	0.179	673	.119
Robbed Someone	0.265**	0.081	0.157†	0.081	−0.053	0.054	0.027	658	.151
Hit in a Fight	0.269**	0.094	0.430***	0.071	−0.051	0.057	0.186	669	.156
Fought at School	0.250**	0.077	0.526***	0.057	−0.058	0.049	0.277	669	.155
Poor Self-Esteem	1.235***	0.111	0.426***	0.029	−0.092**	0.027	0.190	674	.184

^aStandardized estimates from baseline adjusted general linear models using full information maximum likelihood estimation

^bCohen’s d.

^†p < .10.

^*p < .05.

^**p < .10.

^***p < .001.

Discussion

The aim of this test of Mantente REAL was to assess whether this intervention reduced or delayed use of alcohol, tobacco, and marijuana in a community sample of Guatemala City early adolescents, relative to a no-intervention comparison group. We also assessed evidence that the program influenced important antecedents of adolescent substance use behaviors, such as drug resistance skills, permissive attitudes toward drug use, susceptibility to negative peer influences and anti-social behavior, and self-esteem. Students from the schools receiving Mantente REAL reported improvements from pretest to posttest on all these measures, while the comparison group generally changed in undesirable directions. In strict statistical tests, students receiving Mantente REAL had demonstrably better outcomes than the comparison group for cigarette and marijuana use, drug expectancies, actual use of the REAL strategies, and self-esteem.

The intervention group reported relatively better outcomes in areas directly targeted by the intervention: not only substance use behaviors, but also the drug resistance training that is the program’s central mechanism of change, and the preservation of anti-drug attitudes. Some pretest to post-test changes reported by the Guatemala City participants in Mantente REAL were notable compared to other trials of keepin’ it REAL which have shown that it moderates the typical upward trajectories in substance use during adolescence by slowing the rates of initiation and dampening increases in the level of use, but without halting or reversing adolescent substance use in the aggregate (Kulis et al., 2005, 2007). While the comparison group in this trial manifested those upward trajectories, the intervention group showed actual declines from pretest to posttest in recent alcohol, cigarette and marijuana use. In the case of cigarette and marijuana use, the statistically significant intervention effects were produced by modest decreases in the Mantente REAL group and much larger increases in the comparison group. The changes in alcohol use followed the same pattern in direction but the increases in the comparison group were less sizeable than for the other two substances, and were more nearly balanced in size with the decreases reported by the intervention group. The non-significant intervention effect for alcohol may reflect the more pervasive and generally accepted use of alcohol in Guatemala (Kanteres et al., 2009), and greater ability to achieve prevention in adolescence through delaying initiation rather than curbing substance use after it starts. At pretest, the lifetime prevalence of alcohol, cigarette, and marijuana use in the sample was 26%, 11% and 3%, respectively, a pattern consistent with prior estimates for Guatemalan youth (Diaz et al., 1998). It is important to note, however, the marginally significant decline in aggregate alcohol use amounts in the intervention group, suggesting the potential for stronger effects to emerge from a larger trial.

Evidence for another prime target of Mantente REAL—the expansion in the students’ repertoire of effective ways to resist substance use—was strong, exceeding results from the original trial of keepin it REAL (Hecht et al., 2003), where effect sizes for use of the REAL strategies were very small (< .10), half of the effect sizes observed in Guatemala City. The mechanisms of change that Mantente REAL is designed to promote are centered on these drug resistance skills. The curriculum trains youth to acquire a wide repertoire of resistance skills, and promotes communication competence to utilize them appropriately in different contexts. Mastery of drug resistance skills is very important for the large majority of pre-and early adolescents who have not yet started using substances or are only beginning to experiment with use, and lack the skills to reject substance offers in risky situations. The curriculum also teaches risk assessment concerning the consequences of substance use. The impact of the curriculum in this area is reflected in the significant intervention effect for positive expectancies about substance use, which declined among Mantente REAL students while increasing in the comparison group.

Remaining outcomes for which the intervention effects were not statistically significant fell into two groups. Like the results for alcohol use, measures of specific anti-social behaviors and susceptibility to negative peer influences changed in more desirable directions in the intervention than in the comparison group, with estimated effects sizes of about .17. Using G*Power 3.1, a larger sample of about 1,100 respondents would be required for these outcomes to reach statistical significance (p=0.05, power 80%; Faul, Erdfelder, Buchner, & Lang, 2009). The second group includes two attitudinal measures of substance use antecedents: intentions to use substances if given the chance in the coming weekend, and permissive norms towards one’s personal use of substances. In both instances, outcomes changed in desirable directions in both the intervention and comparison groups, for reasons that are not immediately clear. There is a possible developmental explanation: that the somewhat younger target population may have difficulty gauging intentions to use substances reliably in hypothetical scenarios, particularly when they are counterfactual, i.e. when it is unlikely that the opportunity would arise. They also may be more prone, developmentally, to rigid “right versus wrong” thinking and have difficulty calibrating how much they agree or disagree with permissive drug norms.

There are study implications concerning the introduction and transferability of evidence based prevention approaches across borders through international research collaborations, and for larger unresolved debates about a continuum of existing approaches for bringing effective prevention programs to new target populations, from minimal linguistic adaptation of evidence based interventions, to “deep structure” cultural adaptation of EBIs, to the creation of new culturally grounded interventions (Okamoto, Kulis, Marsiglia, Steiker & Dustman, 2014). Several considerations led us to choose the linguistic adaptation strategy for this initial study. First, linguistically adapted versions of kiR were shown to be effective in other Latin American countries. Second, these adapted versions preserve the key intervention components as originally designed, so there is minimal uncertainty about the introduction of new (or elimination of original) intervention elements that may occur with other approaches. Third, the research team from the local non-profit had relationships with Guatemala City schools and had worked in the delivery and evaluation of prevention programs in the USA, which gave them a critical vantage point to assess kiR as applicable for Guatemala City elementary school students. Fourth, linguistic adaptation required minimal resources, suitable for a pilot study budget and the capacities of the implementation sites.

Still, we view this test of the linguistically adapted kiR as appropriate but only an initial step. Although not assured, it was plausible that the original prevention messages in kiR REAL, which were culturally grounded to reflect a multicultural USA middle school population, would resonate as well in Guatemala City, with youth living in violent poor urban neighborhoods, and with participants a year younger than the original target population. Although the original curriculum’s highly interactive lesson format and learning style and the “Americanized” flavor of the examples and videos could have been foreign to the teachers and students, receptivity and enthusiasm were high as indicated by lesson feedback forms that implementing teachers completed after each lesson. During the trial new videos produced in Mexico became available as alternatives to the original USA versions that were dubbed into Spanish. But the implementing teachers in Guatemala reported that students liked the original versions and were interested in comparing their lives to those of Latino students portrayed in the USA videos. Despite these positive indications of the acceptability of the original content of the kiR curriculum to Guatemala students, a subsequent study would be needed to determine whether the cultural influences on youth substance use in Guatemala, and the mix of risk and protective factors shaping such use, differ enough from those in the original kiR target population to recommend a cultural adaptation approach.

Another study implication for delivering evidence based prevention approaches globally is the importance of understanding local capacity for implementation. The local team’s steady presence in the schools throughout the trial and their ongoing solicitation of feedback from principals and teachers identified program delivery barriers quickly. The resources available to schools were limited in ways that made it challenging to implement some aspects of the curriculum as originally designed. Although keepin’ it REAL has been assessed as highly cost-effective in developed countries (Miller & Hendrie, 2008), even the minimal resources needed to implement it may strain schools serving poor communities in Guatemala. For example, the collaborating non-profit organization had to allocate study resources to supply the schools with audio-visual equipment to display the videos that accompanied curriculum lessons. Study schools varied in resources and in size, posing some challenges to consistent and timely delivery of the curriculum lessons, such as when only one teacher served as an implementer at a school. The teachers’ other demands placed limits on the timing and duration of the training that they received. In all the schools, students and teachers were at least indirectly affected by the pervading unstable political climate and atmosphere of chronic resource strain.

Although this trial produced widely consistent evidence of the effectiveness of Mantente REAL in Guatemala City schools, interpretations are limited by the study’s research design and scale. Results are from a non-population-based sample in one metropolitan area, albeit the major city. Schools were recruited from several neighborhoods, but they do not represent all of Guatemala City, other cities, or rural areas of the country. Compared to other trials of keepin’ it REAL with thousands of participants, the smaller scale of this study limited the ability to detect statistically significant intervention effects. Effect sizes for several outcomes were appreciable but would require larger samples to achieve significance. Finally, although it is important to examine a wide array of types of outcomes to assess the impact of the intervention, family-wise Type I errors may have occurred due to the multiple outcomes we examined.

The intervention’s desired impact in Guatemala City on substance use behaviors, attitudinal antecedents of use, and the development of an expanded repertoire of effective drug resistance skills provides promising evidence of the effectiveness of Mantente REAL in a new population, society, and cultural setting. Recommended next steps to confirm the efficacy and appropriateness of Mantente REAL are to conduct larger and more geographically diverse trials; investigate, document and address implementation barriers to wider dissemination within the Guatemala educational system, such as logistical and resource constraints; assess implementer fidelity to the curriculum and the feasibility and acceptability of implementation in a rigorous fashion; and consider subgroup variations in the effectiveness of the curriculum, such as by gender and age. Additional research—and a much higher level of resources—would be needed to determine whether the curriculum’s effectiveness could be further enhanced by a rigorous multi-phase cultural adaptation, including feedback from a wide array of stakeholders (e.g., students, teachers, educational administrators, curriculum experts), a pilot test and further refinement, and subsequent trial comparing effects of the linguistically and culturally adapted versions.