It’s Your Game…Keep It Real in South Carolina: A Group Randomized Trial Evaluating the Replication of an Evidence-Based Adolescent Pregnancy and Sexually Transmitted Infection Prevention Program

Abstract

Objectives. To evaluate the effectiveness of an evidence-based HIV/sexually transmitted infection (STI)/pregnancy prevention program for middle schools implemented by school staff in South Carolina.

Methods. Twenty-four schools, representing 3143 youths, participated in a randomized trial from 2011 to 2014. Students completed surveys before programming (fall of seventh grade), after completing the 2-year It’s Your Game…Keep It Real program (spring of eighth grade), and 1-year postprogram (spring of ninth grade).

Results. There was no statistically significant effect on initiation of vaginal sex between baseline and eighth grade. Significantly fewer students in the comparison condition reported initiating sex at ninth grade, relative to the intervention condition. No group differences existed on other behavioral outcomes that addressed sexual activity in the past 3 months at ninth grade. Seven of 26 psychosocial outcomes (3 knowledge, 1 attitude, 1 self-efficacy, 2 personal limits) were positively affected at eighth grade; 4 remained significant at ninth grade.

Conclusions. The original studies’ behavioral effects were not replicated in this population, possibly as a result of this being an effectiveness trial instead of an efficacy trial, counterfactual exposure design issues, or postprogram exposure to evidence-based programming.

The birthrate among young people aged 15 to 19 years has experienced a long-term decline since the 1990s and is now at an historic low.¹ Despite this success, adolescent birthrates are higher in rural areas than urban areas; in 2010, the adolescent birthrate in rural areas was 43.3 versus 32.7 in urban areas.² This disparity underscores the need for evidence-based sexuality education programs for rural youths. Currently, none of the school-based programs on the US Department of Health and Human Services’ (HHS) list of evidence-based programs (EBP) was developed for rural youths,³ and only a few studies have been published examining the impact of sexuality education programs in rural settings.

Adolescent birthrates in South Carolina are consistently higher than national rates. In 2010, South Carolina had the 12th highest adolescent birthrate in the country, at 42.6 per 1000 compared with 34.2 per 1000 nationally.⁴ Because of these disparities, we applied for and received funding from the replication initiative⁵ of the Office of Adolescent Health to implement and evaluate an EBP in South Carolina middle schools. We selected It’s Your Game…Keep it Real (IYG) from the HHS approved list of programs³ because it was designed for middle school youths and addressed all of health education standards of South Carolina.⁶ We evaluated the effectiveness of IYG in South Carolina middle schools.

IYG has been tested in 2 group randomized trials, both of which showed positive behavioral impacts.^7,8 The first⁷ involved 10 Texas urban middle schools with low-income populations. Results showed that comparison school students were more likely to initiate vaginal, oral, or anal sex by ninth grade than those in intervention schools (P < .05). When addressed separately, results indicated that the intervention delayed oral sex (P < .01) and anal sex (P < .01). IYG also reduced the frequency of vaginal intercourse in the past 3 months (P < .05). In the second study, also in urban Texas,⁸ students in IYG were less likely to initiate any type of sex (P < .01) or vaginal sex (P < .05) relative to students in control schools; they were also less likely to report unprotected vaginal sex in the last 3 months (P < .05) among other favorable behavioral changes.

We describe the implementation and impact of IYG in rural South Carolina middle schools. Based on previous studies, we hypothesized that at follow-up, students in the intervention condition would have lower rates of initiation of vaginal sex, lower rates of recent vaginal sex, lower rates of vaginal sex without protection, and greater levels of psychosocial wellbeing relative to those in the comparison condition.

METHODS

The study, conducted by an external evaluator, involved a group randomized trial in which we randomized schools before baseline to receive the intervention or serve as comparison sites that received standard sexuality education. The evaluator’s institutional review board approved the study. We used a restricted randomization procedure to optimize the balance of characteristics possibly related to sexual initiation, including school configuration and size, anticipated exposure to an EBP in ninth grade, racial/ethnic distribution, free lunch eligibility, and percent of students who passed standardized tests (see the Appendix, available as a supplement to the online version of this article at http://www.ajph.org).

Students were surveyed at baseline before program implementation (fall of seventh grade, 2011), 0 to 6 months after the program ended (spring of eighth grade, 2013), and approximately 12 to 18 months postprogram (spring of ninth grade, 2014). The amount of time between the end of programming and each follow-up survey varied across students because IYG was implemented each semester (spring or fall) using a schedule that worked best for each school. A second round of baseline surveying occurred in February 2012 in 6 comparison schools because of low parent consent return rates.

Recruitment and Study Participants

Twenty-four mainstream public middle schools, all rural, from 10 school districts throughout South Carolina participated for the duration of the study. Schools had to be willing to participate, agree to randomization, serve seventh and eighth graders, have no existing EBP addressing adolescent pregnancy, have Comprehensive Health Education Committee approval—a district level committee required by state law⁶—for the intervention, and agree to not implement a sexuality EBP during the study period if assigned to the comparison condition. Students had to be enrolled in seventh grade at a participating school in the fall of 2011, not have limited capabilities or special needs as determined by the school, and speak English.

To participate in the survey, active parent consent (i.e., opt-in), collected by school staff and student assent, and obtained by trained data collectors, was required. Because this occurred after randomization, the evaluator worked with school staff and data collectors to ensure parents and students were blinded to the experimental conditions in the school during the consent and baseline survey administration process. Each student who returned a completed consent form (regardless of parents’ responses) received a $5.00 gift card.⁹

Timing of student recruitment for the program was left to schools’ discretion; it typically occurred 1 to 3 weeks before implementation. School staff coordinated a passive parent consent (i.e., opt-out) process for participation in the program; this process was separate from the survey participation consent.

Positive consent was obtained for 1775 of 2269 eligible intervention students (78%) and 1469 of 2007 eligible comparison students (73%). Of students with consent, 1725 intervention (97%) and 1418 comparison (97%) students completed the baseline survey. A total of 1496 intervention and 1264 comparison students completed the eighth grade follow-up (86.7% and 89.1%, respectively, of the baseline sample). The ninth grade survey was completed by 1357 intervention and 1130 control students (78.7% and 79.7%, respectively, of the baseline sample). Figure A (available as a supplement to the online version of this article at http://www.ajph.org) summarizes the final analytic samples.

Study Conditions

Intervention.

IYG is a 2-year intervention that consists of twenty-four 50-minute lessons, 12 delivered in seventh grade and 12 delivered in eighth grade, and was developed using a systematic instructional design process to ground its content in social cognitive theory, social influence models, and the theory of triadic influence.⁷ In each grade, the program integrates group-based classroom activities with personalized journaling and individual, tailored, computer-based activities. It also includes 3 parent–child activities in each grade designed to facilitate dialogue on topics including friendship qualities, dating, and sexual behavior.

In this study, IYG was implemented by teachers or staff members employed by the district. Facilitators completed a required 2-day training for each grade level conducted by curriculum developers. We provided technical assistance in real time based on requests and flags in implementation data. Lessons were delivered during regular classroom time (e.g., gym or science class) during the fall or spring semester according to the schedule that worked best at each school, with no more than 2 weeks between lessons. The program replaced previous seventh and eighth grade sexuality content in the 12 intervention schools.

Counterfactual.

Comparison schools provided their usual sexuality education programs, which varied by district, and included activities that addressed some or all of the following topics consistent with state standards⁶: puberty and reproductive health, healthy relationships, decision-making (general and sexual health), communicating values about sex, identifying and avoiding risky situations, adolescent pregnancy, HIV/AIDS/sexually transmitted infections (STIs), abstinence, condoms and contraception, media influences, and dating violence. Although similar to the intervention, no counterfactual program was evidence-based.

Data Collection

Implementation evaluation.

Facilitators completed online implementation logs within 2 days of completing each lesson and submitted attendance data after completing the 12 lessons. Observations, conducted by trained evaluation staff, assessed fidelity and quality of implementation of 5% of lessons. Health teachers at comparison schools and those teaching the study cohort in ninth grade completed online surveys asking about the content of and time spent implementing any sexuality education to the study cohort (Table A, available as a supplement to the online version of this article at http://www.ajph.org).

Impact evaluation.

A student self-report survey provided data for outcome analyses. Trained data collectors administered the survey during school using audio-enhanced, computer-assisted surveys. No monetary incentives were provided, but students were allowed to keep headsets they received for completing the baseline survey. Students received gift cards for follow-up survey completion ($5.00 if completed during class, $10.00 if completed on students’ own time).

Students no longer enrolled in their original study school at follow-up were tracked and surveyed in 1 of several ways, in order of priority: (1) at their current school using the same procedures as in study schools, (2) using an online or mail survey, or (3) using an abbreviated telephone survey.

Measures

The student survey was drawn from Peskin et al.¹⁰ for comparability with previous IYG evaluations. Because of district restrictions, all items that assessed sexual intercourse focused on vaginal intercourse only. We assessed 5 behavioral outcomes: initiation of sex by the (1) eighth and (2) ninth grade follow-ups; (3) at ninth grade follow-up, intercourse in the past 3 months; (4) intercourse without effective birth control in the past 3 months; and (5) for comparability to previous studies, intercourse without a condom in the past 3 months. Initiation indicators were created only for students who reported never having sex at baseline. Students who reported not having sex in the last 3 months or never having it were coded as “No” on questions that addressed recent sexual activity to include the full sample in the analyses.

We assessed 26 psychosocial measures based on the program theoretical model (Table 1). Internal consistency for multi-item scales ranged from a Cronbach’s α of 0.64 to 0.88.

TABLE 1—

Psychosocial Outcomes: Scales and Indexes With Basic Psychometric Properties, Effectiveness of It’s Your Game…Keep It Real, South Carolina, 2011–2014

Constructa	No. Items	Example Item	Response Format	αb
Attitudes and beliefs
General beliefs about waiting to have sex	3	I believe it is okay for people my age to have sexual intercourse with a steady boyfriend or girlfriend.	4-point scale (1 = strongly disagree, 4 = strongly agree)	0.82
Beliefs about waiting until marriage to have sex	3	It is important to me to get married before having sexual intercourse.	4-point scale (1 = strongly disagree, 4 = strongly agree)	0.76
Number of reasons to not have sex	10	Here are some reasons for not having sexual intercourse. Why would you choose NOT to have sexual intercourse at your age? Example reason: I could feel guilty afterward.	Yes = 1, no = 0; count of yes responses	NA
Number of reasons to have sex	9	Here are some reasons for having sexual intercourse. Why would you choose to have sexual intercourse at this age? Example reason: to be popular.	Yes = 1, no = 0; count of yes responses	NA
General beliefs about condoms	3	I believe condoms should always be used if a person my age has sexual intercourse.	4-point scale (1 = strongly disagree, 4 = strongly agree)	0.87
Normative beliefs
Perceived friends’ beliefs about waiting to have sex	3	Most of my friends believe it is okay for people my age to have sexual intercourse with a steady boyfriend or girlfriend.	4-point scale (1 = strongly disagree, 4 = strongly agree)	0.85
Perceived friends’ beliefs about condoms	3	Most of my friends believe condoms should always be used if a person my age has sexual intercourse.	4-point scale (1 = strongly disagree, 4 = strongly agree)	0.88
Perception of number of friends that have a boyfriend or girlfriend	1	How many of your friends have a boyfriend or girlfriend?	5-point scale (1 = none, 5 = all)	NA
Perception of number of friends that have had sex	1	How many of your friends have had sexual intercourse?	5-point scale (1 = none, 5 = all)	NA
Perception of number of peers that have had sex	1	Most teens my age are having sexual intercourse.	4-point scale (1 = strongly disagree, 4 = strongly agree)	NA
Knowledge
General condom knowledge	6	Do condoms help a person keep from getting pregnant?	Yes, no, not sure; count of number correct	NA
General HIV/STI knowledge	3	Some STDs put you at higher risk of getting infected with HIV.	True, false, not sure; count of number correct	NA
Knowledge of signs and symptoms of STIs	6	Mark if you think the following are common signs of having an STD (not including HIV). A headache.	Yes, no; count of number correct	NA
Perceived self-efficacy
To refrain from having sex	6	Imagine you are alone with someone you like very much. Could you stop them if they wanted to touch your private parts below the waist, but you did not want them to?	4-point scale (1 = No, I definitely could not stop them, 4 = Yes, I could definitely stop them)	0.85
To negotiate the use of condoms with a partner	2	Imagine that you and your boyfriend or girlfriend have been having sexual intercourse but have not used condoms. You really want to start using condoms. How sure are you that you could tell your partner you want to start using condoms?	4-point scale (1 = I definitely could not, 4 = I definitely could)	0.75
To obtain and correctly use condoms	3	If you decided to have sexual intercourse, how sure are you that you could have a condom with you when you needed it?	4-point scale (1 = I definitely could not, 4 = I definitely could)	0.64
Personal limits
Know how far I’d go sexually and can communicate it to a partner	1	Look at the sentences below. Mark which one best matches how you feel about what you would do or not do sexually.	4-point scale (1 = I have never really thought about how far I will go sexually, 4 = I know how far I would go sexually and I could tell a partner what I would do or not do)	NA
Know what I think about condom use and can communicate it to a partner	1	Look at the sentences below. Mark which one best matches how you feel about using condoms if you have sexual intercourse.	4-point scale (1 = I have never really thought about using condoms, 4 = I know whether I would use condoms and I could tell my partner)	NA
Intentions
To have sex in the next year if have the chance	1	Do you intend to have sexual intercourse in the next year, if you have the chance?	4-point scale (1 = yes, definitely, 4 = no, definitely not)	NA
To remain abstinent until the end of high school	1	Do you intend to be sexually abstinent (that is, not have sexual intercourse) from now until the end of high school?	4-point scale (1 = yes, definitely, 4 = no, definitely not)	NA
To remain abstinent until marriage	1	Do you intend to be sexually abstinent (that is, not have sexual intercourse) from now until marriage?	4-point scale (1 = yes, definitely, 4 = no, definitely not)	NA
To use a condom if have sex in the next year	1	If you have sexual intercourse in the next year, do you intend to use (or have your partner use) a condom?	4-point scale (1 = yes, definitely, 4 = no, definitely not)	NA
To use effective birth control if have sex in the next year	1	If you were to have sexual intercourse in the next year, do you intend to use (or have your partner use) any of these methods of birth control? [condoms, birth control pills, the shot (Depo-Provera), the patch, the ring (NuvaRing), IUD (Mirena or ParaGard), implant (IMPLANON)]?	4-point scale (1 = yes, definitely, 4 = no, definitely not)	NA
To get tested for HIV/STI if think at risk	2	If you thought you were at risk for having HIV, would you go get tested for HIV?	Yes, no, not sure; count of yes responses	NA
Environmental factors
Communication with parents about sex	6	How many times has your parent/guardian ever talked to you about HIV, AIDS or other sexually transmitted diseases?	3-point scale (1 = we’ve never talked about it, 3 = we’ve talked about it lots of times)	0.82
Exposure to risky situations	5	In the past 3 mo, how often have you invited a boyfriend or girlfriend to your home when an adult was not home?	4-point scale (1 = never, 4 = 6 or more times)	0.74

Note. IUD = intrauterine device; NA = not applicable; STD = sexually transmitted disease; STI = sexually transmitted infection.

^aThe term “sex” means vaginal sex.

^bCronbach’s α based on students’ baseline responses.

The survey included 12 demographic and individual characteristic items drawn from IYG surveys.¹⁰

Analytic Approach

We used multilevel analysis to model data in the presence of anticipated correlation between observations from students within the same school; we modeled the school as a random effect. We used logistic models for dichotomous outcomes and linear models for underlying continuous outcomes. All models used maximum likelihood estimation techniques, specified a random intercept, and were conducted using Stata 13.1 (StataCorp, College Station, TX).¹¹

Each model included the following variables: (1) intervention condition; (2) the baseline outcome (when applicable); (3) age, gender, and race/ethnicity; (4) a set of a priori identified covariates that the literature suggested are related to sexual behaviors¹² and that differed at a P value of less than .15 between conditions, when adjusted for clustering, in the sample of students who completed a baseline and corresponding follow-up survey (Table 2 lists the potential covariates); (5) an indicator that represented when students completed their baseline survey (fall 2011 or February 2012); (6) design variables used in randomization (school enrollment, school configuration, and potential exposure to an EBP in ninth grade); and (7) in behavioral outcome models only, a school-level covariate that represented the percent of students who reported they ever had sex at baseline (excluding those who completed baseline surveys in February). The last was included to control for potential environmental or normative influences that might have resulted from the observed imbalance in rates of reported vaginal sex in the sample of students who completed a baseline survey in the fall (7.4% and 9.0% in the intervention and comparison conditions, respectively). Psychosocial analyses included baseline individual responses to “ever had sex.” We estimated the effect sizes using Cohen’s d or adjusted odds ratios with associated 95% confidence intervals.^13,14

TABLE 2—

Characteristics and Baseline Comparability of the Intervention and Comparison Conditions for Students Completing Eighth Grade Follow-Up and Ninth Grade Follow-Up: Effectiveness of It’s Your Game…Keep It Real, South Carolina, 2011–2014

	Eighth Grade Follow-Up Sample (n = 2760)			Ninth Grade Follow-Up Sample (n = 2487)
Baseline Measure	Intervention % or Mean (SD)	Comparison % or Mean (SD)	B (95% CI)a	Intervention % or Mean (SD)	Comparison % or Mean (SD)	B (95% CI)a
Behavioral outcomesb
Ever had sex (% yes)	6.2	8.9	−0.28 (−0.90, 0.33)	6.5	7.6	−0.09 (−0.65, 0.47)
Had sex in the past 3 mo (% yes)c	3.1	4.0	−0.19 (−0.87, 0.49)	3.1	3.4	−0.06 (−0.70, 0.57)
Had sex without condom in past 3 mo (% yes)d	1.3	1.7	−0.24 (−0.98, 0.50)	1.3	1.4	−0.09 (−0.85, 0.67)
Had sex without effective birth control in past 3 mo (% yes)d	1.2	1.7	−0.36 (−1.04, 0.31)	1.1	1.2	0.001 (−0.84, 0.84)
Demographics
Age, y (mean)	12.7 (0.5)	12.8 (0.5)	−0.14 −(0.22, −0.06)	12.7 (0.5)	12.8 (0.5)	−0.11 (−0.19, −0.04)
Sex (% female)	54.5	51.4	0.11 (−0.07, 0.30)	54.0	52.5	0.06 (−0.13, 0.24)
Race/ethnicity (%)e
Non-Hispanic Black	37.4	40.7	−0.18 (−0.92, 0.56)	36.9	40.4	−0.17 (−0.94, 0.59)
Hispanic/Latino	11.4	11.5	0.05 (−0.79, 0.89)	11.3	10.9	0.00 (−0.75, 0.95)
Non-Hispanic White	39.2	38.3	0.04 (−0.73, 0.82)	40.4	39.6	0.06 (−0.76, 0.89)
Otherf	12.0	9.0	0.30 (−0.01, 0.61)	11.4	9.0	0.27 (−0.03, 0.56)
English is main language spoken in the home (% yes)	88.2	89.1	−0.13 (−0.90, 0.64)	88.3	89.3	−0.16 (−0.92, 0.61)
Live in another home some of the time (% yes)	27.7	28.2	−0.003 (−0.27, 0.27)	27.3	27.3	0.03 (−0.25, 0.31)
Mother was an adolescent parent (% yes)	27.6	34.0	−0.22 (−0.58, 0.15)	26.1	32.4	−0.23 (−0.58, 0.11)
No. of biological parents in the home (%)			0.15 (−0.15, 0.45)			0.16 (−0.14, 0.45)
Neither biological parent	7.0	8.7		7.0	8.2
One biological parent	46.5	49.4		45.5	49.1
Both biological parents	46.5	40.9		47.5	42.7
Academic grades (mean, 1–4)g	3.50 (0.67)	3.34 (0.75)	0.11 (−0.01, 0.22)	3.52 (0.67)	3.37 (0.75)	0.10 (−0.01, 0.22)
Educational aspirations (mean, 1–6)h	5.06 (1.35)	5.00 (1.36)	0.03 (−0.15, 0.20)	5.08 (1.33)	5.04 (1.33)	0.02 (−0.14, 0.18)
Importance of faith (mean, 1–4)i	3.05 (0.88)	3.08 (0.87)	0.03 (−0.19, 0.13)	3.05 (0.87)	3.11 (0.85)	−0.06 (−0.23, 0.11)
No. of times went to a religious service in past 12 mo (mean, 1–6)j	4.36 (1.53)	4.32 (1.56)	0.04 (−0.28, 0.35)	4.36 (1.53)	4.34 (1.55)	−0.001 (−0.33, 0.33)

^aThe coefficients (B) and 95% confidence intervals (CIs) are adjusted for clustering at the level of random assignment.

^bThe baseline rate of sexual initiation in the comparison condition is lower in the ninth grade analytic sample than in the eighth grade analytic sample. Two factors may contribute to this finding. Endpoint analyses were used to address outcomes, therefore the samples did not include the same exact cases for each analysis. Further, although not significant at P < .05, attrition analyses suggest that students who reported ever having sex at baseline were more likely to drop out at higher rates in the comparison condition than the intervention condition (P = .08).

^cStudents who reported never having sex were coded as no.

^dStudents who reported never having sex or not having sex in the past 3 mo were coded as no.

^eDummy variables were used to represent race in the outcome models; baseline equivalence for race was tested separately for each category.

^fOther includes multiple races, American Indian or Alaska Native, Asian, and Native Hawaiian or Pacific Islander.

^gRange is 1 = Mostly Ds and Fs to 4 = Mostly As and Bs.

^hRange is 1 = does not plan to finish high school to 6 = plans to complete a graduate degree.

ⁱRange is 1 = not at all important to 4 = very important.

^jRange is 1 = never to 6 = once a week.

For behavioral outcomes, we conducted 2 sensitivity analyses to understand (1) the influence of the school-level covariate representing the percent of students reporting they ever had sex at baseline, and (2) the influence of including students who completed the later baseline survey.

We adjusted the psychosocial model P values using Bonferroni’s method for multiple testing within domains based on original theoretical constructs (e.g., the “normative beliefs” construct had 5 outcomes, so 5 multiple tests were considered).¹⁵

We used attrition analyses to assess whether any sample characteristics were associated with students who did not complete follow-up surveys and whether the association(s) differed by condition.

RESULTS

Despite strong implementation of the program and positive impacts on some psychosocial outcomes, the anticipated behavioral impacts did not occur. Post-hoc analyses provide data to understand this pattern of results.

Implementation Evaluation

Tables B and C (available as supplements to the online version of this article at http://www.ajph.org) present additional details on sexuality programming provided to students in both conditions.

Intervention condition.

Fidelity and quality of implementation by IYG facilitators was high, as was students’ exposure to the curriculum. Facilitators reported delivering all 12 lessons to every class in both the seventh and eighth grade years. An average of 98% of IYG activities were implemented for an average of 624 minutes of programming in seventh grade and 600 minutes in eighth grade. On a scale ranging from 1 = poor to 5 = excellent, observers rated overall quality of implementation as a 4.5 for seventh grade lessons and as a 4.4 for eighth grade lessons. Students attended an average of 11.4 sessions of 12 in seventh grade and 10.1 sessions of 12 in eighth grade; 11% did not receive eighth grade IYG because of school transfers.

Counterfactual.

Ten of the 12 comparison schools provided sexuality education when the study cohort was in seventh grade, averaging 440 minutes of programming. In eighth grade, 8 schools provided programming, averaging 405 minutes. None implemented an EBP in either year.

Ninth grade programming.

When the study cohort entered ninth grade, most students matriculated into 1 of 20 high schools. Eight high schools reported providing the EBP Safer Choices¹⁶ to the study cohort, which reached approximately 52% of comparison youths and 40% of intervention youths. High schools’ use of Safer Choices was equalized across condition during randomization, but 1 high school that served only students from an intervention middle school did not teach Safer Choices as planned, which resulted in unbalanced exposure to an EBP. The other 12 high schools provided an array of sexuality education; none were evidence-based.

Outcome Evaluation

Attrition analyses.

Intervention condition was not statistically significantly related to participant attrition for either follow-up survey. Youths with certain characteristics—Latino or “other” versus White, who were older, had poorer academic grades, mothers who were adolescent parents, and students who drank more often in the last 30 days—were less likely to complete the eighth and ninth grade surveys than were those who did not report these characteristics (P < .05). However, none of these characteristics showed differential attrition rates between conditions. Differential attrition approached significance for the ninth grade sample on “ever had sex,” which showed that students who reported sex at baseline dropped out at higher rates in the comparison condition than the intervention condition (41% vs 31%; P = .08).

Baseline equivalence.

Except for age, there were no differences between conditions at a P value of less than .05 on baseline measures of demographic characteristics or behavioral variables for either the eighth or ninth grade survey samples (Table 3). Comparison students were, on average, 0.1 years older than intervention students (P < .01), primarily because of the subset that completed the later baseline survey. Table D (available as a supplement to the online version of this article at http://www.ajph.org) shows the baseline equivalence for the analytic samples.

TABLE 3—

Results From Multilevel Logistic Regression Models of Behavioral Outcomes and Intraclass Correlations: Effectiveness of It’s Your Game…Keep It Real, South Carolina, 2011–2014

	Benchmark			Sensitivity 1b		Sensitivity 2c
Outcome variables	No.	AORa (95% CI)	ICC	No.	AORa (95% CI)	No.	AORa (95% CI)
Initiation of vaginal sex
By end of eighth grade	2501	1.18 (0.91, 1.53)	0.002	2501	1.11 (0.74, 1.64)	2256	1.17 (0.90, 1.52)
By end of ninth grade	2268	1.27 (1.01, 1.59)	0.000	2268	1.23 (0.92, 1.64)	2049	1.26 (1.00, 1.58)
Had vaginal sex in past 3 mo
Yes	2392	1.24 (0.94, 1.64)	0.006	2392	1.16 (0.84, 1.61)	2156	1.25 (0.95, 1.66)
Without effective birth control	2392	1.11 (0.77, 1.58)	0.001	2392	1.03 (0.69, 1.52)	2156	1.13 (0.79, 1.61)
Without condom	2387	1.30 (0.93, 1.81)	0.000	2387	1.23 (0.89, 1.72)	2151	1.33 (0.95, 1.86)

Note. AOR = adjusted odds ratio; CI = confidence interval; ICC = intraclass correlation. Estimates were obtained from logistic multilevel models that specified school as a random effect. All models were adjusted for the following school-level design factors: school configuration, enrollment size, and proportion of students who reported having had sex at baseline. All models were also adjusted for the following potential student-level confounders: age, gender, race/ethnicity, number of months between baseline and ninth grade follow-up survey, late enrollment into the study (i.e., January to February 2012), and academic grades. Models addressing recent sexual behaviors also included the student’s baseline measure of the outcome.

^aAn AOR > 1 indicates that more students in the intervention condition reported doing the behavior relative to those in the comparison condition.

^bSensitivity analyses excluding the school-level covariate “ever had sex.”

^cSensitivity analyses excluding the students who completed a baseline survey in February 2012.

Behavioral outcomes.

Behavioral impact results are listed in Table 3. Among students who had not yet had sex at baseline, there was no statistically significant difference between conditions in vaginal sex initiation rates at eighth grade follow-up. At ninth grade follow-up, students in the intervention condition were 1.27 times more likely to initiate sex than were students in the comparison condition (P < .05), which equates to a Cohen’s d of approximately 0.10. There were no statistically significant differences in the likelihood that students from either condition reported engaging in the other behavioral outcomes at ninth grade.

Sensitivity analyses overall supported these results, except for ninth grade sexual initiation. The finding was not confirmed when the model excluded school-level baseline rates of ever having sex, but was confirmed when the model excluded comparison students who completed the baseline survey in February 2012.

Psychosocial outcomes.

After we adjusted for multiple testing within theoretical constructs, at eighth grade follow-up, we found statistically significant differences between conditions in the hypothesized direction for 7 psychosocial outcomes in the areas of knowledge, self-efficacy, and personal limits; 4 remained significant at the ninth grade follow-up (Table 4).

TABLE 4—

Results From Multilevel Linear Regression Models of Psychosocial Outcomes From Eighth Grade and Ninth Grade Follow-Ups, Effectiveness of It’s Your Game…Keep It Real, South Carolina, 2011–2014

	Eighth Grade Follow-Up (n = 2738)			Ninth Grade Follow-Up (n = 2465)
Outcomea,b	No.c	b 95% CI	Estimated Effect Sized	No.c	b 95% CI	Estimated Effect Sized
Attitudes and beliefs
General beliefs about waiting to have sex	2468	0.04 (−0.08, 0.15)	0.03	2227	0.01 (−0.11, 0.07)	0.00
Beliefs about waiting until marriage to have sex	2576	0.08 (−0.02, 0.18)	0.06	2324	−0.02 (−0.10, 0.11)	−0.02
No. of reasons to not have sex	2659	0.32 (0.08, 0.56)	0.10*	2387	−0.02 (−0.34, 0.30)	−0.01
No. of reasons to have sex	2368	–0.03 (−0.13, 0.19)	−0.01	2130	0.08 (−0.28, 0.13)	0.03
General beliefs about condoms	2479	–0.02 (−0.07, 0.03)	−0.04	2229	−0.03 (−0.10, 0.05)	−0.03
Normative beliefs
Perceived friends’ beliefs about waiting to have sex	2568	0.10 (−0.02, 0.21)	0.06	2308	0.03 (−0.07, 0.12)	0.02
Perceived friends’ beliefs about condoms	2410	0.04 (−0.01, 0.10)	0.06	2160	0.00 (−0.06, 0.06)	0.01
Perception of number of friends that have a boyfriend or girlfriend	2665	–0.11 (−0.03, 0.25)	−0.06	2401	−0.03 (−0.07, 0.13)	−0.02
Perception of number of friends that have had sex	2629	–0.02 (−0.14, 0.19)	−0.01	2369	0.01 (−0.17, 0.14)	0.01
Perception of number of peers that have had sex	2648	–0.01 (−0.11, 0.13)	−0.01	2386	0.04 (−0.16, 0.08)	0.03
Knowledge
General condom knowledge	2515	13.06 (9.97, 16.15)	0.33***	2263	5.72 (1.24, 10.20)	0.11*
General HIV/STI knowledge	2409	9.23 (6.56, 11.89)	0.28***	2179	4.68 (1.38, 7.98)	0.12*
Knowledge of signs and symptoms of STIs	2121	3.67 (0.85, 6.49)	0.11*	1928	−0.59 (−3.36, 2.17)	−0.02
Perceived self-efficacy
To refrain from having sex	2471	0.01 (−0.06, 0.09)	0.02	2229	0.01 (−0.07, 0.08)	0.01
To negotiate the use of condoms with a partner	2447	0.00 (−0.05, 0.04)	0.00	2192	−0.02 (−0.08, 0.04)	−0.02
To obtain and correctly use condoms	2357	0.23 (0.14, 0.32)	0.21***	2141	0.13 (0.06, 0.19)	0.16***
Personal limits
Know how far I’d go sexually and can communicate it to a partner	2473	0.29 (0.19, 0.40)	0.22***	2238	0.18 (0.08, 0.28)	0.15***
Know what I think about condom use and can communicate it to a partner	2427	0.12 (0.04, 0.19)	0.13**	2193	0.01 (−0.08, 0.09)	0.01
Intentions
To have sex in the next y if have the chance	2644	0.04 (−0.17, 0.09)	0.02	275	0.05 (−0.19, 0.08)	0.03
To remain abstinent until the end of high school	2599	0.05 (−0.09, 0.19)	0.03	2343	0.08 (−0.07, 0.23)	0.04
To remain abstinent until marriage	2599	0.09 (−0.08, 0.25)	0.04	2337	0.11 (−0.02, 0.23)	0.07
To use a condom if have sex in the next y	2546	–0.06 (−0.11, 0.01)	−0.09	2283	−0.07 (−0.12, −0.01)	−0.10
To use effective birth control if have sex in the next y	2494	0.05 (0.00, 0.10)	0.08	2240	−0.01 (−0.06, 0.04)	−0.02
To get tested for HIV/STI if think at risk	2392	0.00 (−0.07, 0.06)	0.00	2157	−0.03 (−0.09, 0.02)	−0.05
Environmental factors
Communication with parents about sex	2399	0.01 (−0.03, 0.05)	0.03	2152	0.00 (−0.05, 0.05)	0.00
Exposure to risky situations	2452	0.00 (−0.06, 0.07)	0.00	2298	−0.05 (−0.03, 0.13)	−0.05

Note. CI = confidence interval; STI = sexually transmitted infection. Estimates were obtained from linear multilevel regression models that specified school as the random effect, All models were adjusted for the following school-level design factors: school configuration and enrollment size. In addition, models were adjusted for the following student-level covariates: age, gender, race/ethnicity, baseline measure of the outcome, number of months between baseline and follow-up survey, late enrollment into the study (i.e., January to February 2012), ever had sex at baseline, and academic grades. The P values have been adjusted for multiple testing within groupings of variables based on the original theoretical constructs. Attitudes and beliefs adjusts for 5 tests; normative beliefs adjusts for 5 tests; knowledge adjusts for 3 tests; perceived self-efficacy adjusts for 3 tests; personal limits adjusts for 2 tests; intentions adjusts for 6 tests; and environmental factors adjusts for 2 tests.

^aAll psychosocial outcomes were coded as protective (i.e., higher values and a positive Cohen’s d are in the same direction as program goals).

^bThe term “sex” means vaginal sex.

^cSample sizes vary because of missing data. For consistency with previous It’s Your Game studies, scale scores were computed only if a respondent answered all of a scale’s questions.

^dEstimated effect size was computed using Cohen’s d, for which the formula is where t is the t-statistic and df is the degrees of freedom.¹⁴ Because Cohen’s d is a transformation of the coefficient and its SE, P values apply to both the coefficient estimate and the effect size.

^*P < .05.

^**P < .01.

^***P < .001.

Exploratory analyses.

We used post hoc multilevel-layered analyses, a series of successive regression analyses, to explore the contribution of groups of imbalanced baseline covariates to the ninth grade initiation finding after controlling for design and standard demographic indicators (Table E, available as a supplement to the online version of this article at http://www.ajph.org). We entered the variable sets in the following order: (1) condition, (2) design factors, (3) late enrollment, (4) student demographic characteristics, (5) other student characteristics, (6) school-level baseline rates of initiation, and (7) school-level exposure to an EBP. When only condition was entered, sexual initiation rates in the intervention and comparison conditions were equivalent (25.9% and 25.0%, respectively; P = .79). With each model adjustment, the intervention rate remained relatively steady, whereas the comparison rate dropped. A near-significant difference emerged in step 6 (P = .053), and it became statistically significant in the last step (intervention = 25.4%, comparison = 21.1%; P = .04).

Analysis of dosage data among intervention students only (Table F, available as a supplement to the online version of this article at http://www.ajph.org) indicated that more programming exposure was related to a reduced likelihood of reporting sexual initiation by eighth grade follow-up (adjusted odds ratio = 0.94; P = .03). This finding did not hold for behaviors measured at ninth grade follow-up.

DISCUSSION

We tested a high-quality implementation of IYG compared with the usual sexuality education in South Carolina. However, behavioral results did not replicate previous studies.^7,8 Rates of sexual initiation at the end of eighth grade did not differ between conditions, nor did any of the ninth grade outcomes that addressed risky sexual behavior in the past 3 months. The only statistically significant difference was in the opposite direction of hypothesized effects, which suggested that usual programming outperformed IYG, although the magnitude of the difference was small.

Our study and the original studies^7,8 showed modest impacts on psychosocial outcomes; however, there were fewer statistically significant findings in our study. Earlier studies showed positive impacts on intentions and self-efficacy related to delaying or refusing sex and intentions to use condoms; these effects were not replicated in our study. Refusal self-efficacy significantly mediated the effect of IYG on sexual initiation using data from the first trial,¹⁷ which suggests its importance.

Both study context and design issues might have contributed to our pattern of findings. First, our study was an effectiveness trial that used classroom teachers for implementation rather than an efficacy trial more tightly controlled by the original researchers; existing literature suggested effectiveness trials often yield smaller effects than efficacy trials.¹⁸ Second, the program was tested in an alternate geographic region (SC vs TX) and setting (rural vs urban) with different racial/ethnic compositions (primary subgroups included youths who identified as Black or White in SC vs Black or Latino in TX); this might have affected the relevancy of lessons for youths.¹⁹ In addition, regional or cultural differences did not allow for questions about oral or anal sex to be included in the South Carolina survey, both of which were positively affected in previous IYG studies.^7,8 Finally, students in both conditions received sexuality education on similar topics throughout the study period—approximately 7 hours per year in comparison schools and 10 hours per year in intervention schools—minimizing any comparative difference between conditions. This was not the case in the original studies, in which students in the comparison condition reportedly received minimal programming (4–6 hours total; personal oral communication, Susan Tortolero Emery, PhD, Prevention Research Center, School of Public Health, University of Texas-Houston, September 16, 2015).²⁰ One plausible interpretation was that IYG was not as effective as other local programming on the measured behaviors.

Despite randomization, 2 important imbalances between conditions emerged. First, an imbalance in school-wide rates of vaginal sex at baseline could represent environmental or normative factors capable of influencing rates of initiation. The sensitivity analysis excluding this factor did not confirm the results from the benchmark analysis, which raised concerns regarding the robustness of benchmark estimates. Second, a greater proportion of comparison students received an EBP in ninth grade than did the proportion of intervention students. Layered exploratory analyses demonstrated that rates of sexual initiation at ninth grade follow-up were equivalent until several covariates were entered into the model; a significant difference emerged only when these 2 factors were included. In addition, differential attrition might have contributed to the pattern of results for some outcomes (e.g., unprotected sex), particularly when compounded with an imbalanced exposure to an EBP in ninth grade.

Finally, exploratory dosage analyses suggested that the more lessons youths attended, the less likely they were to initiate sex by eighth grade follow-up. This difference was no longer statistically significant at the end of ninth grade, but underscores the importance of program exposure to maximize effectiveness.

Strengths and Limitations

Our study was unique in its focus on replicating an EBP in a rural setting. Despite the strong design—randomization of schools and long-term follow-up—some imbalances occurred, which affected the interpretation of results. The study included youths in rural middle schools; results might not generalize to urban regions.

Conclusions

Behavioral effects found in the original studies in urban Texas were not replicated in this rural South Carolina population. Because our study varied on multiple factors during implementation, including setting, population, and outcome, it was difficult to obtain the effects of any single factor, although counterfactual, sensitivity, and exploratory analyses suggested the impact of some effects. Further replication research will continue to explore how these factors affect the robustness of original findings.

Our study also highlighted the need to develop and test other programming options with middle-school aged youths in rural settings. No school-based EBPs were developed and evaluated for rural settings,³ and some evidence suggests that rural youths might respond differently to programs initially developed and tested in urban settings.¹⁹ This might be a result of a mismatch in underlying theoretical models,¹⁹ a mismatch in content or context (e.g., failing to address the most influential risk and protective factors related to sexual behaviors in that region), or other factors, such as unaddressed cultural or regional differences. Mediation analyses could help identify which program components are a better fit for rural communities and which may need modification. Data from this and other replication studies can help guide practitioners in making better program selections and provide considerations for future development of sexuality interventions in rural settings.