Teachers’ patterns of implementation of an evidence-based intervention and student outcomes: results from a nationwide dissemination over 24-months follow-up

Abstract

More information is needed about factors influencing real-life implementation and program impact of interventions effective in controlled study conditions. Ongoing national implementation of an evidence-based HIV prevention program targeting grade 6 students in The Bahamas offers the opportunity to examine patterns of implementation and relate them to student outcomes. Data were collected from 208 grade 6 teachers, 75 grade 7 teachers and 4,411 grade 6 students followed over two years. Mixed-effects modeling analysis examined the association of teachers’ patterns of implementation with student outcomes. High quality program implementation in grade 6 (high implementation dosage and fidelity) was significantly related to student outcomes six and 18 months post-intervention. Quality of implementation of the booster session in grade 7 was also significantly related to student outcomes in grade 7. Quality of delivery of the brief booster session a year after initial implementation is important in maintaining or resetting the student outcome trajectory.

Introduction

Within the past decade, there have been significant advances in HIV/AIDS treatment and prevention options [1]. Globally, the importance of anti-retroviral therapy to reduce mother-to-infant transmission [2] and transmission between sexual partners [3,4] as well as post-exposure prophylaxis for needle-stick injuries [5] and high-risk sexual exposure [6,7] are well recognized. Circumcision is widely acknowledged to reduce transmission from an infected to an uninfected individual [8-10]. The march towards a successful HIV-vaccine has become compellingly close [11]. Both needle exchange and opioid substitution [12,13] programs and improved approaches to blood transfusion [14,15] have been shown to reduce HIV transmission. In 2013, 2.1 million new HIV infections were reported, 38% lower than those reported in 2001. AIDS-related deaths were 35% lower in 2013 compared to 2005. Globally a steadily increasing number of persons have access to treatment, although this number remains below 50% of those infected [16]. Appropriately, these important scientific breakthroughs and public health achievements have been widely heralded [1,17-20] and have led to more nuanced approaches of combination prevention [18,19,21] and/or treatment as prevention [22]. However, as impressive as these advances and opportunities are, there is widespread recognition of the growing need for identifying and addressing the issues involved in maintaining the effectiveness of these programs as they move from research to the public health arena. [1, 17, 18, 23-25].

For an intervention which has been demonstrated to be effective in a research setting to become an effective public health tool, it must: a) continue to be delivered (“sustainability”); and, b) be delivered in a fashion that is likely to reproduce its effectiveness (“integrity”). These concepts, long recognized to be important, have been the source of substantial scientific inquiry and have yielded some valuable findings [26, 27]. Research to-date indicates that sustained implementation of evidence-based behavioral interventions is low [28]. Studies assessing fidelity of implementation of effective programs (using a range of definitions, criteria and cutoffs) report considerable deviance from the original curricula [29]. Sustained implementation and fidelity thereof are both overlapping and competitive. Many factors are associated with both sustained implementation and fidelity of implementation (for example, ease of integrating the new program into an existing curricular approach/routine [27, 30]). However, other studies have found that as these interventions are implemented over time and in new locales, some adaptation is inevitable and may signal commitment to the new program [29, 31-33] and/or strengthen the program’s effectiveness [34].

These finding resonate with calls for a focus on the local implementation community as critical to the next stage of HIV research:

“…implementers need to know their epidemic, and to respond in at least three dimensions….HIV rates and behaviors at a local level…the state of a HIV/AIDS spending and programming in relation to the levels of need…[and] an appreciation of macro-level political, demographic, and economic changes..” [19, p.845].

As well:

“…intervention programs must continuously find the balance between the original, effective script but change to reflect with differences in the micro-environment and changes over time in the microenvironment, including economic, scientific and disease characteristics…”[23]

In short, to begin to reap the maximum benefits from the scientific advances made to date and to continue to build upon these breakthroughs to lead to new discoveries, focused attention is needed on their implementation.

Implementation of evidence-based interventions has been identified as a major research priority [26]. While progress has been made in identifying critical factors for continued success of an effective program, many questions remain. Priority areas include addressing the need to adapt the intervention to be responsive to local factors of importance in new settings (e.g., culture, HIV prevalence) and to macro-changes affecting populations in multiple settings (e.g., development of effective new biomedical innovations). These and other changes occur continuously over time, suggesting that to remain effective for new generations of populations, the intervention itself must be adapted over time. Further, other interventions and changes will be occurring that may interact with the intervention of interest and impact its effectiveness either positively or negatively. This complex interplay complicates assessment of the long-term continued effectiveness of interventions “brought to scale”. Nevertheless, important information can be derived from the natural history of community-based intervention implementation by following populations exposed to the intervention, especially if the fidelity of implementation and the initial impact of the intervention on the target audience are known.

Following these populations over time can help us to understand the relative importance of variations in intervention delivery and how these variations impact intervention effectiveness over time in a natural setting. In this manuscript we examine grade 7 youth who, as part of national implementation across all government grade 6 classes in The Bahamas, had received a theory-based behavioral intervention which had previously been adapted in that country and shown through a longitudinal randomized trial to be effective [35-37].

Relevant background to the current study

In the mid-1990’s, The Commonwealth of The Bahamas suffered from an HIV seroprevalence exceeding 4%. Over the past two decades, The Bahamas has devoted great effort and resources to reduce this rate to 2% [38-40]. Central to their approach has been multinational, interagency and multidisciplinary involvement including multiple external agencies and the Bahamian Ministries of Health and of Education. Among the joint endeavors of this partnership has been the adaptation and effectiveness evaluation through two randomized, controlled longitudinal trials of an intervention from the Centers for Disease Control and Prevention’s “Diffusion of Effective Behavioral Interventions” portfolio , Focus on Youth with Informed Parents and Children Together (FOY with ImPACT) (https://www.effectiveinterventions.org/en/AboutDebi.aspx). The resultant program, “Focus on Youth in the Caribbean (FOYC)”, is a sexual risk reduction program targeting mid-adolescents accompanied by “Caribbean Informed Parents and Children together (CImPACT)”, a parental monitoring and parent-adolescent communication video-intervention followed by a discussion. FOYC contains role plays, games, discussions, stories, lectures, exercises and homework assignments to reinforce main messages and increase knowledge, intentions and skills regarding effective communication, negotiation, sexual risk avoidance, and condom use. A decision-making model is applied throughout to illustrate the steps that should guide a child in selecting a course of action. Intervention evaluations showed that the intervention significantly increased Bahamian youth’s HIV/AIDS knowledge, perceptions of their ability to use condoms and condom use intention [36, 37] with evidence of increased condom use [35].

National Implementation of FOYC

Central to the current research, beginning in 2011 all grade 6 Bahamian youth attending one of the nation’s 80 government elementary (primary) schools were exposed to the FOYC curriculum [41]. The FOYC curriculum (consisting of eight sessions, each averaging 45 minutes to 1.5 hours in length) was integrated into the Health and Family Life Education (HFLE) curriculum and taught by grade six teachers throughout the school year. In 2012 through 2014, the former grade 6 youth, now in grade 7, attending government middle schools, received a one hour booster session reviewing decision-making tools, condom use skills and communication skills. Per standard MOE procedure, both the grade 6 and grade 7 teachers were invited to participate in teacher training workshops; for the grade 6 teachers the training lasted for one to four days, for the grade 7, the training was one day. Details of the training provided to the teachers are described in our recent publications [41, 42].

Prior findings relevant to the current research questions

As reported previously [42], cluster analysis was used to identify teachers’ implementation patterns based on implementation dose (i.e., number of core activities completed) and fidelity of implementation (i.e., percentage of core activities being changed during the implementation). Results of the two-step cluster analysis indicated three distinct clusters of teachers: 1) High Implementation Group (32% of the teachers), characterized by high levels of sustained implementation and fidelity of implementation. On average, teachers in this group taught over 80% of core activities (25 out of 30 core activities) and changed only 14% of the core activities; 2) Moderate Implementation Group (53%), showing moderate levels of sustained implementation but high levels of fidelity of implementation. On average, teachers in this group taught less than half of the core activities (12.3 out of 30 core activities) and changed 13% of core activities that they taught in the classroom; and 3) Low Implementation Group (16%), with low levels of sustained implementation and fidelity of implementation. On average, these teachers taught less than one-third of core activities (8.6 out of 30 core activities) and changed three-quarters of the core activities that they taught in the classroom.

Study questions to be addressed

In our analyses conducted at the end of grade 6, student outcomes varied by the quality of implementation of their teachers; students taught by teachers in the Low Implementation Group demonstrated poorer outcomes in HIV/AIDS knowledge, preventive reproductive health skills and intention to use protection if they were to have sex) [42]. In this manuscript we seek to understand: 1) whether the youth continue to show improvement in HIV prevention skills over time (one year after the conclusion of the prevention intervention); 2) whether a brief national FOYC-booster program administered to all grade 7 students changes the outcome trajectory seen among grade 6 students based on grade 6 teacher FOYC- implementation patterns; and 3) whether differing implementation patterns among grade 7 students exerts an independent effect on student outcomes.

Methods

Study site

The research described in this manuscript was conducted in grade 6 classrooms in the 80 government elementary (primary) schools throughout and in grade 7 classrooms in the 34 government middle (junior high) schools throughout The Bahamas. The research protocol was approved by the Wayne State University Human Investigation Committee and the Institutional Review Board of the Bahamian Princess Margaret Hospital, Public Hospitals Authority.

Measures

Sustained implementation and fidelity of implementation

To assess implementation, all teachers were asked to complete a Teacher Implementation Checklist specific for each of the eight sessions of FOYC after they had taught the session. For grade 6 teachers, the checklist includes all 46 activities in the FOYC curriculum, 30 of which were identified by the developers as “core elements”. For the grade 7 teachers, the Checklist includes five activities (all of which were “core” activities) including the fictional family illustrating the long-term impact of decisions made as an adolescent, the decision making model, communication skills, steps in effective condom use and a story about non-consensual intercourse. The teachers indicated which activities they had and had not taught in each session; sustained implementation was defined as the number of core activities that were taught. For those core activities that they taught, the grade 6 teachers also recorded whether they had modified the format of the activity as outlined in the manual to determine fidelity of implementation.

Student outcomes

Using the Bahamian Youth Health Risk Behavioral Inventory (BYHRBI) [36] as a template, the MOE developed a curricular assessment instrument (with identifying information only at the level of the school and teacher) which they administered to grade 6 and grade 7 students (baseline and follow-up in grade 6 and one year later in grade 7). The instrument assessed HIV/AIDS knowledge, preventive reproductive health skills, and relevant perceptions, intentions, and self-reported behaviors. A scale of 15 true/false statements assessed level of HIV/AIDS knowledge (Cronbach’s α=0.71). Correct responses were scored “1” and incorrect “0” (summary scores ranged from 0 to 15 per student). Preventive reproductive health skills were assessed through an adaptation of the Condom-use Skills Checklist [43], a validated scale consisting of true and false statements describing the steps of correct condom use (Cronbach’s α=0.71). Correct responses were scored “1” and incorrect “0”, yielding a summary score of 0 to 6 per student. Self-efficacy for using reproductive health prevention methods was assessed through three items (yes=1/no=0), yielding a possible summary score per student of 0 to 3 per student. The internal consistency of the scale was 0.81. Intention to use protection was assessed by student response to the question “if you were to have sex in the next six months, how likely is it that you would use a condom (to prevent yourself from getting HIV)?” along a five-point Likert scale (1=very unlikely to 5=very likely).

Analysis to address the three study questions

Do youth continue to show improvement in HIV prevention factors one year after the conclusion of the prevention intervention?

The difference in students’ HIV/AIDS knowledge, reproductive health skills, self-efficacy and intention to use condom protection (“HIV-prevention factors”) across the three time points (baseline and grade 6 and grade 7 follow-ups) were examined using ANOVA with the Tukey honestly significant difference (HSD) tests, using the whole sample and stratified by teachers’ patterns of implementation (high/moderate/low implementers). The difference in knowledge, skills, self-efficacy and intention across the three implementation groups at each time point was assessed using ANOVA. In addition, increases in knowledge, skills, self-efficacy and intention from baseline to grade 7 follow-up among the three teachers’ implementation groups were compared using ANOVA.

Do differences in groups based on grade 6 teacher implementation patterns that were apparent six months after FOYC was delivered in grade 6 persist 12 months later?

As described earlier in this manuscript, previously we have shown that grade 6 students taught by the one-third of teachers who were “high implementers” (characterized by high levels of sustained implementation and fidelity of implementation) and the one-half of teachers who were “moderate implementers” (characterized moderate levels of sustained implementation but high levels of fidelity of implementation) performed significantly better than students taught by the one-sixth of teachers who were “low implementers” (demonstrating poor implementation and fidelity [42]). To assess whether these differences in student performance with regard to reproductive health knowledge, skills, self-efficacy and intentions as a function of grade 6 teacher performance grouping persisted one year later, students within the three teacher groupings were compared using ANOVA with the Tukey HSD post hoc tests. We further examined the association of grade 6 teachers’ patterns of implementation (high/moderate/low implementers) with grade 7 student outcomes using mixed-effects modeling, adjusting for clustering effects of classroom and/or school. Independent variables included teachers’ implementation clusters, student’s age, sex, and baseline student outcomes. School and class were included as random effect variables in the model.

Does a brief national FOYC-booster program administered to all grade 7 students change the outcome trajectory established in grade 6 based on grade 6 teacher FOYC- implementation patterns?

To examine whether grade 7 teacher delivery of the booster session impacted student performance, students’ knowledge, skills, self-efficacy and intentions were compared according to their grade 7 teachers’ levels of implementation of the booster session (categorized into three groups: taught none or one activity, two or three activities or four or five of the total of five activities in the booster session) using ANOVA with the Tukey HSD post hoc tests, using whole sample and stratified by grade 6 teachers’ patterns of implementation (high/moderate/low implementers). The effect of the booster session on student outcomes was further assessed using mixed effects models, controlling for clustering effects of classroom and/or school and baseline difference.

To examine the effects of both the grade 6 teachers’ implementation of the FOYC intervention and the grade 7 teachers’ delivery of the booster session on student outcomes, we expanded the above-mentioned mixed effects models by including both grade 6 and grade 7 teachers’ patterns of implementation in the same model. We further examined the relationship of different combinations/scenarios of grade 6 implementation of FOYC (high/moderate/low implementers) and grade 7 teachers’ delivery of booster session (none or one; two or three; four or five activities) with student outcomes using mixed-effects models controlling for clustering effects and baseline difference. All analyses were performed using SAS 9.4 statistical software package (SAS Institute Inc., Cary, NC, USA).

Results

General Description of Students and Teachers

Data were collected from 208 grade 6 teachers, 75 grade 7 teachers and 4,470 grade 6 students at baseline; from 4,183 (93.6%) students at the grade 6 follow-up; and, from 3,439 (76.9%) students at the grade 7 follow-up. The average age of the students at baseline was 10.4 (SD=1.7) years. The vast majority of students are of African descent and about one-half are female.

Do youth continue to show improvement in HIV prevention skills one year after the conclusion of the prevention intervention?

Table 1 presents the changes in HIV/AIDS knowledge, preventive reproductive health skills, self-efficacy and intention to use protection from baseline to grade 6 and grade 7 follow-ups, overall and stratified by grade 6 teachers’ patterns of implementation (high/moderate/low implementers). Overall, student outcomes increased significantly from baseline to grade 6 follow-up, and from grade 6 follow-up to grade 7 follow-up (knowledge: 8.2 vs. 9.9 vs. 10.1, F=718.59, P<0.001; skills: 3.4 vs. 3.8 vs. 4.2, F=391.46, P<0.001; self-efficacy: 0.8 vs. 1.2 vs. 1.6, F=525.51, P<0.001; intention: 2.4 vs. 3.2 vs. 3.8, F=631.08, P<0.001). When stratified by teachers’ implementation group, students’ knowledge, skills, self-efficacy and intention increased significantly across the three time points for all three implementation groups except that the increases in knowledge from grade 6 follow-up to grade 7 follow-up were not significantly different between the high and moderate implementation groups.

Table 1.

Change in student outcomes from baseline to grade six and grade 7 follow-ups, overall and by teacher’s implementation clusters

Variables	Baseline (1)	Grade 6 follow-up (2)	Grade 7 follow-up (3)	Increase (3-1)	F	Post-hoc comparison
Sample size (n)	(4411)	(4168)	(3439)
HIV/AIDS knowledge  (range 0-15 points)	8.22 (2.62)	9.95 (2.55)	10.13 (2.39)	1.91 (2.52)	718.59***	(1,2) (1,3) (2,3)
a) High Implementation Group	8.59 (2.50)	10.45 (2.24)	10.39 (2.30)	1.80 (2.41)	281.59***	(1,2) (1,3)
b) Moderate Implementation Group	8.18 (2.58)	9.98 (2.52)	10.13 (2.37)	1.95 (2.49)	379.02***	(1,2) (1,3)
c) Low Implementation Group	7.83 (2.59)	9.17 (2.66)	9.62 (2.57)	1.79 (2.58)	71.18***	(1,2) (1,3) (2,3)
F	21.75***	58.21***	16.89***	1.575
Preventive reproductive health skills  (range 0-6 points)	3.37 (1.31)	3.77 (1.27)	4.19 (1.19)	0.82 (1.26)	391.46***	(1,2) (1,3) (2,3)
a) High Implementation Group	3.54 (1.28)	3.98 (1.17)	4.26 (1.14)	0.72 (1.22)	117.95***	(1,2) (1,3) (2,3)
b) Moderate Implementation Group	3.38 (1.30)	3.73 (1.29)	4.17 (1.20)	0.80 (1.25)	177.98***	(1,2) (1,3) (2,3)
c) Low Implementation Group	3.07 (1.33)	3.56 (1.36)	4.09 (1.25)	1.02 (1.30)	73.04***	(1,2) (1,3) (2,3)
F	27.09***	27.13***	3.80*	8.93***
Self-efficacy (range 0-3 points)	0.78 (1.05)	1.23 (1.17)	1.62 (1.16)	0.83 (1.10)	525.51***	(1,2) (1,3) (2,3)
a) High Implementation Group	0.79 (1.04)	1.32 (1.16)	1.69 (1.14)	0.90 (1.08)	216.78***	(1,2) (1,3) (2,3)
b) Moderate Implementation Group	0.80 (1.05)	1.26 (1.19)	1.57 (1.16)	0.77 (1.10)	220.19***	(1,2) (1,3) (2,3)
c) Low Implementation Group	0.77 (1.09)	1.08 (1.11)	1.62 (1.21)	0.85 (1.14)	72.49***	(1,2) (1,3) (2,3)
F	0.22	9.17***	3.76*	4.95**
Intention to use protection  (range 1-5 points)	2.38 (1.73)	3.17 (1.79)	3.81 (1.66)	1.42 (1.70)	631.08***	(1,2) (1,3) (2,3)
a) High Implementation Group	2.47 (1.77)	3.27 (1.77)	3.86 (1.62)	1.39 (1.71)	204.42***	(1,2) (1,3) (2,3)
b) Moderate Implementation Group	2.35 (1.72)	3.22 (1.79)	3.84 (1.66)	1.48 (1.69)	344.43***	(1,2) (1,3) (2,3)
c) Low Implementation Group	2.39 (1.73)	2.87 (1.82)	3.54 (1.74)	1.15 (1.73)	50.84***	(1,2) (1,3) (2,3)
F	1.96	10.43***	6.48**	6.24**

Note:

^*P<0.05;

^**P<0.01;

^***P<0.001.

HIV/AIDS knowledge and reproductive health skills were significantly higher in both the grade 6 and the grade 7 follow-ups among students taught by grade 6 high and/or moderate implementation teachers compared to those students taught by grade 6 low implementation teachers. Comparable at baseline, self-efficacy and intention were higher among students with grade 6 high and moderate implementation teachers than those with grade 6 low implementation teachers at the grade 6 and grade 7 follow-ups. The increase in self-efficacy from baseline to grade 7 follow-up for the grade 6 high implementation group was greater than that of the moderate and low implementation groups (0.9 vs. 0.8 vs. 0.8, F=4.95, P<0.01). The increase in intention from baseline to grade 7 follow-up for the grade 6 high and moderate implementation groups was greater than that of the low implementation groups (1.4 vs. 1.5 vs. 1.2, F=6.24, P<0.01) (Table 1).

Do differences in groups based on grade 6 teacher implementation patterns that were apparent six months after FOYC was delivered in grade 6 persist 12 months later?

The results of the mixed-effects models indicate that the quality of grade 6 teachers’ implementation was significantly related to student improvement in HIV/AIDS knowledge and intention to use protection. The relationship of the students’ reproductive health skills with teacher implementation was marginally significant. Changes in self-efficacy were not related to grade 6 teacher implementation. In summary, youth whose teachers belonged to the high and moderate implementation groups demonstrated better student outcomes in grade 7 than those youth whose teachers were in the low implementation group (Table 3).

Table 3.

Mixed-effects models assessing the association between grade 6 teacher’s pattern of implementation and students’ outcomes in grade 7

Variables	Estimated models
	HIV/AIDS knowledge			Preventive reproductive health skills			Self-efficacy			Intention to use protection
	B	SE	t	β	SE	t	β	SE	t	β	SE	t
Fixed effect
Intercept	9.809	0.325	30.14***	4.285	0.342	12.52***	1.579	0.141	11.19***	3.534	0.207	17.04***
Age	−0.011	0.022	−0.49	−0.021	0.030	−0.70	−0.005	0.011	−0.43	0.002	0.016	0.14
Gender
Male	−0.100	0.085	−1.18	−0.019	0.046	−0.42	0.021	0.043	0.49	−0.010	0.063	−0.16
Female (ref)
Baseline student outcome	−0.003	0.017	−0.14	0.013	0.018	0.72	0.038	0.021	1.84#	−0.013	0.018	−0.69
Teacher’s implementation clusters
High Implementation Group	0.755	0.214	3.53***	0.172	0.100	1.72#	0.098	0.090	1.09	0.336	0.139	2.42*
Moderate Implementation Group	0.502	0.204	2.46*	0.064	0.095	0.67	0.013	0.086	0.15	0.330	0.133	2.48*
Low Implementation Group
Random effect
School†	--	--		0.026	0.017	1.51#	0.016	0.012	1.41#	--	--
Class (nested within school)†	0.417	0.080	5.18***	0.025	0.015	1.61*	0.021	0.013	1.61#	0.133	0.033	4.06***

^#P<0.10;

^*P<0.05;

** P<0.01;

^***P<0.001.

^†z test.

Does a brief national FOYC-booster program administered to all grade 7 students change the outcome trajectory established in grade 6 based on grade 6 teacher FOYC- implementation patterns?

Students’ HIV/AIDS knowledge, reproductive health skills, self-efficacy and intention to use protection were significantly associated with grade 7 teachers’ level of implementation of the booster session (Table 2). Knowledge, skills and intention were the highest among youth whose teachers covered four or five activities (“good” implementation) of booster session, followed by youth whose teachers covered two or three activities (“fair” implementation), with youth whose teachers taught none or one activity (“poor” implementation) of the booster session demonstrating the lowest performance. Self-efficacy was higher among youth whose teachers exhibited fair implementation compared to youth whose teachers exhibited poor implementation of the booster session. In stratified analysis among youth whose grade 6 teachers were in the low implementation group, those whose grade 7 teachers exhibited good and/or fair implementation of the booster compared to those whose grade 7 teachers exhibited poor implementation demonstrated greater increases in all student outcomes in grade 7 (knowledge: 10.2 vs. 9.6 vs. 8.7, F=7.15, P<0.001; skills: 4.2 vs. 4.2 vs. 3.5, F=8.99, P<0.001; self-efficacy: 1.8 vs. 1.7 vs. 1.1, F=8.21, P<0.001; intention: 3.9 vs. 3.5 vs. 3.1, F=4.10, P<0.05). Among youth whose teachers were in the grade 6 high implementation group, those whose grade 7 teachers exhibited poor booster implementation compared to those grade 7 teachers who exhibited good and/or fair implementation demonstrated poorer reproductive health skills and intention to use condom protection (skills: 4.0 vs. 4.2 vs. 4.4, F=7.16, P<0.001; intention: 3.4 vs. 3.8 vs. 4.0, F=3.84, P<0.05).

Table 2.

Associations of grade 7 teachers’ implementation of booster sessions with student outcomes

Variables	Number of activities in the booster session completed			F	Post-hoc comparison
	0~1 activities (1)	2~3 activities (2)	4~5 activities (3)
Sample size (n) †	262	1,946	1,014
HIV/AIDS knowledge  (range 0-15 points)	9.66 (2.80)	10.01 (2.42)	10.47 (2.18)	17.65***	(1,3) (2,3)
High implementation group	10.56 (2.34)	10.16 (2.32)	10.63 (2.19)	5.76**	(2,3)
Moderate implementation group	9.89 (2.61)	10.08 (2.46)	10.33 (2.14)	2.39
Low implementation group	8.74 (3.10)	9.55 (2.45)	10.24 (2.32)	7.15***	(1,3)
Preventive reproductive health skills  (range 0-6 points)	3.83 (1.33)	4.15 (1.21)	4.33 (1.13)	19.25***	(1,2) (1,3) (2,3)
High implementation group	4.04 (1.24)	4.16 (1.15)	4.41 (1.12)	7.16***	(1,3) (2,3)
Moderate implementation group	4.03 (1.24)	4.13 (1.24)	4.28 (1.15)	2.94*	(1,3)
Low implementation group	3.47 (1.40)	4.18 (1.23)	4.18 (1.10)	8.99***	(1,2) (1,3)
Self-efficacy (range 0-3 points)	1.46 (1.23)	1.67 (1.15)	1.55 (1.18)	5.95**	(1,2) (2,3)
High implementation group	1.68 (1.21)	1.71 (1.11)	1.67 (1.19)	0.16
Moderate implementation group	1.55 (1.21)	1.64 (1.15)	1.39 (1.17)	7.86***	(2,3)
Low implementation group	1.07 (1.22)	1.69 (1.22)	1.80 (1.12)	8.21***	(1,2) (1,3)
Intention to use protection  (range 1-5 points)	3.53 (1.76)	3.77 (1.68)	3.91 (1.60)	5.62**	(1,3)
High implementation group	3.41 (1.77)	3.83 (1.64)	3.97 (1.57)	3.84*	(1,3)
Moderate implementation group	3.98 (1.62)	3.80 (1.68)	3.84 (1.63)	0.56
Low implementation group	3.06 (1.82)	3.47 (1.75)	3.87 (1.64)	4.10*	(1,3)

Note:

^†217 youth who did not identify their grade 7 teachers were excluded.

^*P<0.05;

^**P<0.01;

^***P<0.001.

When grade 6 teachers’ implementation group (high/moderate/low implementers) and grade 7 teachers’ level of implementation of the booster session (good/fair/poor) were examined simultaneously using mixed-effects modeling, implementation in both years (FOYC in grade 6 and the booster session in grade 7) was significantly related to increased HIV/AIDS knowledge and intention to use protection. Grade 7 teacher’s level of implementation of the booster session was significantly related to increases in reproductive health skills (Table 4).

Table 4.

Mixed-effects models assessing the effects of grade 6 teacher’s implementation of FOYC intervention and grade 7 teacher’s delivery of booster session on student outcomes

Variables	Estimated models
	HIV/AIDS knowledge			Preventive reproductive health skills			Self-efficacy			Intention to use protection
	B	SE	t	β	SE	t	β	SE	t	β	SE	t
Fixed effect
Intercept	9.506	0.366	25.99***	3.985	0.360	11.07***	1.491	0.162	9.22***	2.720	0.563	4.83***
Age	−0.010	0.022	−0.42	−0.023	0.030	−0.74	−0.005	0.011	−0.47	0.057	0.044	1.28
Gender
Male	−0.075	0.088	−0.85	−0.015	0.047	−0.32	0.029	0.045	0.65	−0.049	0.066	−0.75
Female (ref)
Baseline student outcome	−0.003	0.018	−0.17	0.015	0.018	0.83	0.031	0.021	1.42	−0.015	0.019	−0.78
Grade 6 teacher’s implementation  clusters
High Implementation Group	0.711	0.210	3.38***	0.120	0.099	1.22	0.113	0.091	1.25	0.324	0.147	2.20*
Moderate Implementation Group	0.521	0.202	2.59**	0.043	0.095	0.45	0.005	0.087	0.06	0.300	0.142	2.12*
Low Implementation Group
Grade 7 teacher’s level of  implementation of booster session
4~5 activities completed	0.488	0.215	2.27*	0.441	0.109	4.04***	0.000	0.102	0.00	0.309	0.158	1.95*
2~3 activities completed	0.207	0.203	1.02	0.320	0.103	3.11**	0.161	0.096	1.67#	0.117	0.150	0.78
0~1 activities completed (ref)
Random effect
School†				0.017	0.015	1.12	0.019	0.012	1.58#	0.003	0.031	0.08
Class (nested within school)†	0.357	0.077	4.61***	0.025	0.016	1.50#	0.014	0.013	1.12	0.142	0.045	3.18***

^#P<0.10;

^*P<0.05;

^**P<0.01;

^***P<0.001.

^†z test.

Table 5 presents the relationship of different combinations of grade 6 implementation of FOYC and grade 7 delivery of the booster session with student outcomes. Compared to youth whose grade 6 teachers belonged to the high implementation group and whose grade 7 teachers covered four or five activities of the booster session (i.e., good implementation), youth whose grade 6 teachers belonged to low implementation group and whose grade 7 teachers taught none or one activity of the booster session (i.e., poor implementation) performed more poorly in all four student outcomes. Youth whose grade 6 teachers belonged to the moderate implementation group and whose grade 7 teachers exhibited poor or fair implementation of the booster session performed more poorly in HIV/AIDS knowledge and reproductive health skills; youth whose grade 6 teachers belonged to the high implementation group but whose grade 7 teachers exhibited poor implementation of the booster session had poorer reproductive health skills and intention to use condom protection. Importantly, youth whose grade 6 teachers belonged to low implementation group but whose grade 7 teachers demonstrated good implementation of the booster session (i.e., four or five activities) demonstrated similar student outcomes in comparison to the highest performance group.

Table 5.

Mixed-effects models assessing the effects of different combinations of grade 6 teacher’s implementation of FOYC intervention and grade 7 teacher’s delivery of booster session on student outcomes

Variables	Estimated models
	HIV/AIDS knowledge			Preventive reproductive health skills			Self-efficacy			Intention to use protection
	B	SE	t	β	SE	t	β	SE	t	β	SE	t
Fixed effect
Intercept	10.694	0.321	33.33***	4.584	0.353	12.99***	1.670	0.138	12.12***	4.029	0.200	20.11***
Age	−0.009	0.022	−0.42	−0.024	0.030	−0.79	−0.005	0.011	−0.48	0.002	0.016	0.13
Gender
Male	−0.073	0.088	−0.83	−0.015	0.047	−0.32	0.033	0.045	0.72	−0.013	0.065	−0.20
Female (ref)
Baseline student outcome  Nine combinations of grade 6  implementation of FOYC and grade  7 booster delivery	−0.001	0.018	−0.08	0.016	0.018	0.88	0.030	0.021	1.41	−0.016	0.019	−0.84
Low implementation of FOYC and   poor delivery of booster	−1.415	0.390	−3.63***	−0.887	0.191	−4.63***	−0.489	0.180	−2.71**	−0.898	0.282	−3.18**
Low implementation of FOYC and   fair delivery of booster	−1.005	0.253	−3.98***	−0.155	0.125	−1.25	0.038	0.115	0.33	−0.599	0.173	−3.47***
Low implementation of FOYC and   good delivery of booster	−0.491	0.355	−1.38	−0.222	0.182	−1.22	−0.087	0.171	−0.51	0.081	0.254	0.32
Moderate implementation of FOYC   and poor delivery of booster	−0.832	0.343	−2.42*	−0.362	0.171	−2.11*	−0.026	0.159	−0.16	0.194	0.240	0.81
Moderate implementation of FOYC   and fair delivery of booster	−0.440	0.184	−2.39*	−0.243	0.091	−2.68**	−0.001	0.084	−0.01	−0.218	0.126	−1.73#
Moderate implementation of FOYC   and good delivery of booster	−0.231	0.202	−1.14	−0.112	0.099	−1.13	−0.241	0.091	−2.64**	−0.124	0.140	−0.89
High implementation of FOYC and   poor delivery of booster	−0.097	0.371	−0.26	−0.394	0.181	−2.18*	0.035	0.167	0.21	−0.537	0.264	−2.03*
High implementation of FOYC and   fair delivery of booster	−0.340	0.176	−1.93*	−0.174	0.090	−1.92*	0.043	0.085	0.51	−0.200	0.125	−1.60
High implementation of FOYC and   good delivery of booster (ref)
Random effect
School†	-	-		0.015	0.014	1.03	0.017	0.011	1.51#	0.002	0.024	0.10
Class (nested within school)†	0.342	0.078	4.38***	0.023	0.016	1.39#	0.011	0.012	0.90	0.119	0.038	3.10***

^#P<0.10;

^*P<0.05;

^**P<0.01;

^***P<0.001.

^†z test. Good delivery of booster=covered 4-5 activities; fair delivery of booster=covered 2-3 activities; poor delivery of booster=covered 0-1 activity.

Discussion

The present study examined how patterns of teacher implementation (as measured by high/moderate/low implementation clusters) of an effective intervention program and the subsequent implementation of a booster session relate to student outcomes over 18-months follow-up within the context of nationwide dissemination of an evidence-based intervention (FOYC). The availability of multiple indicators of implementation quality (number of activities and sessions completed and number of core activities being changed) and multiple student outcomes provided a unique opportunity to explore these associations. The findings indicate that high quality implementation (characterized by high levels of implementation and fidelity of implementation) was significantly related to three of the four student outcomes (including HIV/AIDS knowledge, reproductive health skills and intention to use protection) at six months after intervention implementation. The quality of implementation also predicted greater increases in HIV/AIDS knowledge and intention to use protection 18 months after the initial implementation (after controlling for the implementation of booster session in grade 7). The level of implementation of the booster session in grade 7 was significantly related to three student outcomes (after controlling for initial implementation of FOYC in grade 6). There was a dose-outcome relation in terms of the activities covered in the booster session, suggesting that the proportion of content delivered is important to maintain desired program outcomes after the initial implementation. These findings highlight the importance of implementation quality of the initial delivery and subsequent booster session in achieving program goals.

This pattern of results is largely consistent with those from earlier, randomized, controlled longitudinal trials of similar interventions (FOYC+CImPACT) showing student benefits in knowledge, skills, perceptions and intention [44, 45]. In previous effectiveness trials, the FOYC intervention significantly increased youths’ perceptions of their ability to use condoms [46]. However, in this national implementation study, self-efficacy increased significantly from baseline to the grade 6 follow-up and from grade 6 to the grade 7 follow-up for all student participants. The increase in self-efficacy was not associated with teachers’ level of implementation. It is possible that this lack of association with quality of delivery represents “intervention decay” which has been described in other studies during broad dissemination [46] or it may result from the external reinforcement of condom-use throughout The Bahamas conducted as part of their overall national HIV prevention strategy [39]. It is also possible that the modified brief version of the three-item self-efficacy measure (e.g., “I could get condoms”) is less sensitive to implementation effects, within the context of a nationwide scale-up effort.

Data in the present study indicate that the national implementation of FOYC has been successful in improving students’ HIV/AIDS knowledge, reproductive health skills and intention to use condom protection. The success of implementation varied within and across schools, with some teachers implementing the intervention with higher quality than others. This variation may be influenced by a combination of individual characteristics and institutional factors [47]. Due to the significant variation between schools and/or classrooms with regard to implementing the FOYC program, students’ exposure to the intervention curriculum and improvement in knowledge, skill and intention varied considerably, as indicated by significant classroom/school random effects in the mixed-effects models. In the schools and/or classes where the implementation quality was poor, students demonstrated diminished gains in comparison to youth in other schools and classes with regard to program-related outcomes at six months post-implementation.

An important and exciting finding in this study was that good delivery of the booster session containing five “core” activities in grade 7 allowed significant improvement of the poor student outcomes resulting from weak implementation of FOYC by the grade 6 teachers. Substantial increases in all student outcomes were observed among youth whose grade 6 teachers had been in the low implementation group but whose grade 7 teachers taught 4-5 activities of the booster session. Conversely, a poor delivery of booster session in grade 7 can undermine the positive impact of high quality of grade 6 implementation (especially reproductive health skills and intention to use condom protection).

There are several potential limitations in this study. First, all the measures used in this analysis were based on teachers’ and students’ self-reports. It is possible that teachers over-reported their level of implementation of the intervention curriculum [48]. In the current study, trained observers independently observed and assessed approximately 20% of each teacher’s classes. We compared the teacher and observer reports on activities taught in these sessions and found that the teacher-observer agreement was high (over 80%), indicating that teacher’s self-reports are reliable [41]. Second, there was relatively high attrition rate (i.e., 30%) to grade 7 follow-up. The major reasons for attrition in grade 7 included students’ non-identification of their grade 6 teachers in the grade 7 follow-up survey (n=217) and loss of contact due to students’ graduation from primary school and transferring into non-government middle schools (private or religious-based schools in The Bahamas or private schools in other countries such as the USA). Third, information on individual students’ attendance at lessons were not collected as the implementation targeted at all students in classrooms. Student-level exposure to the intervention was not controlled for in the analysis.

Despite these limitations, this study adds to our understanding about the effects of large-scale implementation of effective interventions in real-world condition by examining patterns of implementation over time and relating them to student outcomes. Our results support the existing literature showing that enhanced implementation of a prevention program is associated with more positive program effects [49]. Importantly, the brief booster session delivered one year after the initial intervention demonstrated significant effects on almost all student outcomes, suggesting that a booster session is important to maintain the long-term program effects. Moreover, the quality of booster implementation can significantly impact the student performance trajectory, indicating the critical importance of teacher training and booster delivery after the initial intervention delivery.