Preparing for National Implementation of an Evidence-based, Effective HIV Prevention Program among Bahamian Sixth-Grade Students

Abstract

Using data from the preparatory phase prior to national implementation of an effective HIV prevention program [Focus on Youth in the Caribbean (FOYC)] in all Bahamian government sixth-grade classes, we describe: 1) actual FOYC implementation; 2) factors which influenced implementation; and 3) the relationship of implementation with intervention outcome. Six elementary schools (with 17 grade six classrooms) were selected to participate in the preparatory phase. The 17 teachers were invited to: attend a training workshop; coordinate administration of questionnaires to the students; teach the 10 sessions of FOYC; and complete self-assessment checklists. 395 students submitted baseline and 311 students submitted year-end questionnaires. Thirteen teachers initiated FOYC; five completed all 10 sessions. Implementation of FOYC was not related to teacher FOYC workshop experience but did cluster by school. There were significant positive correlations between improved student knowledge of HIV/AIDS, protective health skills, perceived parental monitoring and reduced risk behaviors with the number of FOYC sessions delivered. Implementation was impeded by logistic impediments, structural issues with the measures, and comfort-level issues, most of which can be addressed for national implementation. Degree of FOYC implementation is correlated with positive student outcomes.

Introduction

The field of human immunodeficiency virus (HIV) prevention has advanced considerably over the last several decades. Forty-one “Best Evidence” effective HIV prevention programs have been identified by the CDC's Prevention Synthesis Project (CDC 2009). Our increased understanding of the processes of diffusion and dissemination of effective behavioral interventions has led to an increased recognition of the need to focus on the related but unique issues of implementation of effective interventions (Green et al. 2009; Dearing 2008). Towards this end, another program within the CDC, the “Diffusion of Effective Behavioral Interventions” (DEBI) project, selected “best evidence programs” and packaged them to enable communities to implement them in a fashion most likely to retain their effectiveness (CDC 2008; Lyles et al. 2007; Dworkin et al. 2008).

One of the DEBI programs is “Focus on Youth with Informed Parents and Children Together” (FOY with ImPACT). As the DEBI project is committed to implementing effective programs in areas most likely to benefit from effective HIV prevention efforts, the US researchers who developed FOY with ImPACT have been collaborating with researchers from the Bahamian Ministry of Health (MOH) and Ministry of Education (MOE) to adapt the intervention for use in Bahamian government schools.

It has been reported that The Bahamas, with an estimated population of 353,658 individuals on over 700 islands, has the highest estimated adult HIV prevalence rate (3%) in the Caribbean (UNAIDS 2010). Heterosexual activity is the predominant mode of transmission. The Bahamas has focused considerable resources towards decreasing the epidemic; in 2010 there were only 59 AIDS-related deaths in The Bahamas, the lowest number since the beginning of the epidemic. However, the epidemic has been especially prominent among young adults; 57% of non-AIDS HIV cases are among individuals ages 15 to 34 years, who represent < 20% of the population. HIV is the leading cause of death among young adults ages 25 to 44 years in the Bahamas. (Maura 2011; UNAIDS 2010; MOH 2011).

For more than ten years, the US-Bahamian team evaluated the Bahamian adaptation of FOY [a 10-session (plus two booster sessions) adolescent HIV prevention intervention entitled “Focus on Youth in The Caribbean” (FOYC) and a 1-hour parental monitoring intervention entitled “Caribbean Informed Parents and Children Together” (CImPACT)]. A randomized, controlled three-celled effectiveness trial involving 15 elementary schools in The Bahamas was conducted. Based on the effectiveness of the intervention through 36 months, the MOE decided to incorporate FOYC into the sixth-grade curriculum within the government school system (Deveaux et al. 2011).

The need to move science to practice has inspired countless publications and serves as the foundation for the NIH Roadmap Initiative (Zerhouni 2006; Zerhouni 2007). The principle focus of implementation science is on the process of delivery and how this process impacts expected outcomes (based on prior effectiveness research) (Dearing 2009). Implementation science may occur in an experimental setting but may also be conducted during natural or real-life implementation. National implementation of an evidence-based behavioral intervention offers a unique opportunity to contribute to the emerging field of behavioral implementation science (Rogers 1995; McKleroy et al. 2006). Given the scope of a national implementation effort, the process generally entails several phases. Implementation research from one phase may be used to inform implementation efforts in a subsequent phase.

Factors influencing fidelity of implementation

Fidelity of behavioral implementation refers to the degree to which program implementers (“trainers”, or in the current situation, grade six teachers) implement programs and implement them as intended by the program developer (in this case, the developers of FOYC) (Dearing 2009). Dating back at least to the mid-1970's with the Rand Report entitled “Implementation of Educational Innovation” (Berman and McLaughlin 1976), concerns have been raised regarding the fidelity of implementation; the consensus of multiple evaluations of implementation of education and behavioral interventions is that some degree of alteration is unavoidable (Rohrbach et al. 2005; Ringwalt et al. 2004; Bell et al. 2007; Galbraith et al. 2009; Hallfors and Godette 2002; Dusenbury et al. 2003; Bellg et al. 2004). In describing the literature regarding implementation of behavioral interventions, Rohrbach et al (2005) observed:

Previous research has shown that decisions to adopt, or initiate the use of, innovative programs and practices in schools are often made by school boards, and upper- and mid-level school district administrators.…In early stages of the process, teachers may give their opinions about the proposed program, but in general they are not the ultimate decision makers about whether the program will be adopted across the school district or within the school. However, when it comes to program implementation, most teachers have considerable autonomy regarding what goes on in their own classroom. Thus, once a school district has adopted a new curriculum, it is typically the teachers who decide whether and how it is actually used.” ^{page 515}

Ringwalt and colleagues (2004) reported that among a representative sample of US teachers, 79.8% reported adapting their prevention curricula.

The NIH Behavior Change Consortium has identified five areas to consider in conceptualizing treatment fidelity in conducting research: study design; training providers; delivery of treatment; receipt of treatment; and enactment of treatment skills (Bellg et al. 2004). These factors can be grouped according to those which potentially could be impacted during the training process, those that may be affected by fixed characteristics of the trainers (teachers) or the respondents, and those associated with the intervention itself.

Factors which could be modified during training

Potentially intervenable characteristics that have been identified as associated with increased fidelity include those that potentially increase trainer confidence, perceived relevance, ownership and relative benefit. Those factors that could be altered and might result in increased confidence include: 1) intensive in-service training that follows a set curriculum (compared to no training, training that is ad hoc or variable in nature, non-participatory training and/or brief training) (Berman and McLaughlin 1976; Dusenbury et al. 2003; Perrin et al. 2006; Kelly et al. 2000); 2) less curricular discretion (e.g. a more detailed script or lesson plan) (Dusenbury et al. 2003; Perrin et al. 2006); and 3) practice in skills necessary to teach the curriculum and to use interactive methods (Fixsen et al. 2005). Those factors that potentially result in increased relevance include a perception that the focus of the intervention applies to the community in which the intervention is being adopted (Rohrbach et al. 1993). The extent to which relevance is more local (as opposed to a global or national relevance), this factor may be a stronger motivator of fidelity. Related to relevance, ownership appears to be important; (specifically, a sense on the part of the trainees that the intervention addresses a local issue and that they or their community had significant input into the development of the intervention). Also relevant to the question of benefit is the perception of the effectiveness of currently available (in this case HIV prevention) curricula; if the existing curriculum is perceived as adequate, substitution with a new one is likely to be more difficult (Dusenbury et al. 2003).

Other characteristics of the teachers that may influence fidelity

Characteristics of the teacher and teaching environment have also been shown to increase fidelity. Examples include shorter duration of time as a teacher, initial feelings about the existing curriculum (in this case the existing HIV prevention in the schools), and an over-all positive attitude toward prevention in general (Dusenbury et al. 2003; Parcel et al. 1995). Likewise, organizational characteristics including receptivity to innovation and decreased turmoil have been associated with increased fidelity (Wandersman et al. 1988; Gottfredson 1984). Evidence that local authority figures (e.g. the school principal and guidance counselors) support the importance of the intervention being adopted serves to indicate the relative benefit to be derived from teaching this curriculum as opposed to more time spent on another subject (such as math or reading) (Dusenbury et al. 2003).

Factors associated with the intervention that may influence fidelity

Complex programs are less likely to be successfully implemented; as a corollary, programs that are packaged and present the materials in a straight-forward manner are more faithfully executed (Dusenbury et al. 2003). Detailed instruction manuals for the trainers increase fidelity (Bauman et al. 1991).

Evidence that fidelity is associated with sustained effectiveness

One of the most comprehensive examinations of fidelity and effectiveness to date was conducted by Blakely et al. (1987) assessing seven nationally disseminated education and criminal justice projects. Results indicated that implementations conducted with high-fidelity were more effective than low-fidelity implementations. They measured program fidelity as the proportion of finite program components that were implemented and developed detailed process measures that allowed observation of specific program components to measure adoption or implementation by each site to determine fidelity. Multiple methods, including extensive interviews and in person observations were utilized. Validity was analyzed by comparing data across the various measures obtained. Multiple sources of data were available for 75% of items at each site. They found an exact-agreement convergence rating of .96 for the between-source comparison strategy summed across all 70 sites. They found that high-fidelity adopters tended to be more effective than implementers with low-fidelity, local additions to the model tended to enhance effectiveness, and modifications not distracting from fidelity were unrelated to effectiveness.

As significant as this research is, there are several reasons why additional research on the relationship between fidelity of HIV prevention intervention delivery and HIV risk reduction outcomes is still critically important. As the researchers themselves describe, the outcomes data were left up to the sites to describe and provide and thus were inconsistent in nature and of variable quality. The procedure to examine the relationship between outcomes and fidelity was limited to a ranking procedure and the relationship itself was suggestive rather than definitive. Finally, both the content and the target audiences of the interventions assessed in the Blakely study differ from HIV prevention among early adolescents.

In summary, the national implementation of FOYC throughout the sixth-grade classes among all 78 of the government primary schools in The Bahamas affords HIV prevention efforts a unique opportunity to explore issues arising in the implementation of an effective intervention. This is also an opportunity to increase our understanding of factors affecting implementation and to test the relationship between implementation and replication of intervention outcomes. The data and observations acquired during the preparatory phase described herein offer guidance both for the wider Bahamian national implementation effort to follow and for the general field of implementation science.

Materials and Methods

In the spring of 2010, the Ministry of Education (MOE) selected six elementary schools to participate in the first phase of national implementation. Four of the schools were located in New Providence (NP) (the four schools are heretofore referred to as NPA, NPB, NPC, and NPD) and two on one of the neighboring islands, locally referred to as the Family Islands (heretofore referred to as FI).

The six selected schools housed 17 sixth-grade classes and teachers: NPA, five teachers; NPB, four teachers; NPC, three teachers; NPD, three teachers; and, the two schools from the FI each had one teacher. NPB and NPC had participated in the original sixth-grade effectiveness study and had been randomized to FOYC; NPD had also participated in the prior study but had been randomized to the control condition. NPA and the two NI schools had not been in the prior study.

The research protocol was approved by the Wayne State University Human Investigation Committee and the Institutional Review Board of the Bahamian Public Hospitals Authority.

Guiding theoretical model

The national HIV prevention intervention in the Bahamas is guided by social cognitive theory, which provides a conceptual framework for understanding human behavioral change. The theory posits that human behavior is the product of the dynamic interplay of personal factors, behavior, and environmental influences (Bandura 1986). Behavior can be changed through new learning experiences, guidance in protective perceptions, and support for the development of capacities (Glanz et al. 2008). In the context of HIV prevention and sexual risk reduction, prevention knowledge and skills to exercise self-protective behavior are necessary but not sufficient for behavior change. Behavioral change is mediated by a process of cognitive appraisal by which people integrate knowledge and outcome expectancies from a judgment of their ability to perform the protective behavior (e.g., practice abstinence). Change in HIV-related behaviors results from the joint functions of increased HIV prevention knowledge and skills, parental monitoring, and well-established self-efficacy. The model has been empirically confirmed in multiple populations and communities (CDC 1999; O' Leary et al. 2000).

Teacher Training

The MOE established a teacher training workshop format following the protocol utilized during the effectiveness trial of FOYC and used by the DEBI program to train future interventionists in the delivery of FOY. The teacher training covered: 1) a review of the need for HIV prevention in The Bahamas; 2) an overview of FOYC including research showing its effectiveness; 3) a walk-through of each of the 10 sessions of FOYC with full participation and “teach-backs” of the activities considered to be critical to the success of FOYC; and 4) a didactic question-and-answer period regarding methods of protection. All 17 teachers (regardless of attendance at a workshop) were given a copy of the FOYC teacher training manual.

Five of the schools (both of the FI schools and three of the four NP schools) were represented at the training by seven teachers (see Table 1). Among the 10 remaining teachers scheduled to participate in the FOYC national implementation, six had attended a prior FOYC workshop and participated in the original FOYC training as part of the effectiveness evaluation; four received no training.

Table 1. Preparation for teaching Focus on Youth in the Caribbean (FOYC) among the 17 sixth-grade teachers in Phase One of national implementation.

Individual schools	Number of teachers who were:
	Present at FOYC Workshop	Received prior FOYC Training	Received no FOYC Training
New Providence (NP) Schools
NPB*	2	1	1
NPD*	1	1	1
NPC+	0	1	2
NPA	2	1	2
Family Island (FI) Schools
#FI#1	1	0	0
#FI#2	1	0	0

^*School was part of original FOYC study and was randomized to receive FOYC

⁺School was part of original FOYC study and was randomized to receive the control condition

Two team members (US and Bahamian) were present for observations. Detailed notes were maintained regarding each activity covered during the training.

Expectations of the teachers in the classrooms

The teachers were told by the MOE that they would be expected to a) coordinate the administration of an anonymous baseline knowledge, skills and practice questionnaire to their sixth-grade students at the beginning of the school year (prior to teaching FOYC) and again at the end of the year; b) teach all 10 sessions of FOYC; c) complete a self-assessment checklist after delivering each session and permit observer assessments as described in next section; d) complete a pre-and post implementation questionnaire assessing factors contributing to implementation and fidelity (see next section); and e) communicate with project staff regarding completion or lack thereof of these activities.

Self-assessments and Observer assessments

A critical component of implementation research involves assessing whether the curriculum was taught and the extent to which it was consistent with the training components and with the form of delivery that was originally found to be effective. Implementation “fidelity” was to be assessed by both a teacher self-assessment, and independent observer assessments of two of the 10 sessions taught by each teacher. These assessments were conducted using a pre-formatted “check list” (yes/no and multiple choice format) and a written explanation of observations/self-reflections. These forms were derived from the measures successfully used in the prior FOYC effectiveness trial and used in the DEBI roll-out of FOY.

In order to assess the factors described earlier in this manuscript that prior research has suggested may influence likelihood of implementation (Berman and McLaughlin 1976; Dusenbury et al. 2003; Perrin et al. 2006; Bauman et al. 1991), each teacher was asked to complete a 14-item pre- and post-implementation survey regarding their perceptions of the importance and appropriateness of the contents of FOYC for sixth-grade youth as well as in their ability to teach it, importance of the curriculum compared to other topics, comfort-level with FOYC, etc. Finally, a brief, one-on-one open-ended interview was conducted with each teacher regarding their experiences with implementation (or lack thereof) of FOYC.

Student Questionnaires

To assess student response to the intervention, a shortened version the Bahamian Youth Health Risk Behavioral Inventory (BYRHB) (Deveaux et al. 2011; Chen et al. in press) was administered at the beginning of sixth-grade before delivery of FOYC and the end of sixth-grade, in the classroom by the sixth-grade teachers (or their surrogates). The questionnaires were anonymous and confidential. An 18-item scale including true and false statements was used to assess level of knowledge regarding HIV/AIDS. Correct responses were scored 1 and incorrect 0, resulting in a mean score of 0-18 for each participant. Knowledge of specific skills regarding protected sex were assessed using a published skills checklist scale, whose internal consistency, construct validity, and criterion-related validity have been validated through comparison of responses to a knowledge questionnaire and an actual demonstration of condom-use skills (Stanton et al. 2009).The checklist scale includes correct steps and incorrect steps; correct responses were scored “1” and incorrect “0

A six items-scale was used to assess self-efficacy regarding protected sex; agreement was measured through a five-point Likert scale (1=strongly disagree to 5=strongly agree). Intentions/expectations regarding several risk and protective behaviors (including sexual intercourse) in the next six months were assessed by self-reported likelihood (1=very unlikely to 5=very likely). For each of these categories, a mean score was derived using individuals' responses to the items, with higher scores indicating higher levels of knowledge of protective skills, self-efficacy and behavioral intentions respectively. The students were asked if they had engaged in certain risk behaviors (including sexual intercourse in the last six months). Youth were administered Silverberg's six-item Parental Monitoring Score in which youth perception of parental awareness of their activities along a five-point Likert scale (1=never to 5=always) was assessed (Silverberg and Small, 1991).

Analysis

The pre-intervention/post-intervention student self-report data were entered into IBM SPSS Statistics 19. After data cleaning, the data were converted to SAS format. All the statistical analyses were performed using SAS 9.2 statistical software package (SAS Institute Inc., Cary, NC, USA).

Descriptive statistics (mean and standard deviation) of knowledge and attitudes and frequency distribution of demographics, intentions and behaviors were calculated. Baseline comparisons were made between schools and between sexes to examine potentially significant differences at baseline, using ANOVA and student's t test. When ANOVA results indicate statistical significance, post-hoc comparisons, using Tukey's HSD test, were made to determine which groups differed. Similar analyses were conducted for the end-of school year questionnaires collected. Using student's t test and Pearson's χ² test, pre-intervention and post-intervention data were compared overall and by the number of educational sessions taught by each teacher.

To adjust for the clustering effects (classroom and/or school), the intraclass correlation coefficient (ICC) and variance inflation factor (VIF) were calculated for all outcome variables including knowledge, attitudes, intention and behaviors (Wears 2002). ICC values ranged from 0 to 0.03. All test statistics (t, F, χ²) were adjusted using VIFs; corresponding p values to the adjusted test statistics are displayed in Tables 2 and 3.

Table 2. Teachers' assessment pre- and post-implementation of the Focus on Youth in the Caribbean (FOYC) curriculum.

Teachers' assessment	Pre-implementation	Post-implementation
Total number of teachers	5	9
Number (%) of teachers endorsing the following statements:
*Prevention programs are very important for youth	5 (100%)	9 (100%)
*HIV prevention is very important for youth in general	5 (100%)	9 (100%)
*HIV prevention is very important for grade 6 youth in general	5 (100%)	7 (77.8%)
*HIV prevention is very important for the Bahamian community	5 (100%)	9 (100%)
*“Focus on Youth” is very important for the grade 6 youth in our school	4 (80%)	7 (77.8%)
*The time I spent teaching “Focus on Youth” is about the same compared to the time I spent teaching reading skills in grade 6	4 (80%)	7 (77.8%)
*I feel comfortable in teaching the materials in “Focus on Youth”	3 (60%)	NA
*Grade 6 students benefited very much from “Focus on Youth”	NA	6 (66.7%)

Note: NA=not available

Table 3. Baseline knowledge and perceptions regarding risk and protective factors among sixth-grade students in The Bahamas.

Variable	Overall	Primary schools				adj. F	Post-hoc test	Gender		adj.t
		NPB (1)	NPD (2)	NPC (3)	NPA (4)			Male	Female
Sample size (n)	395	99	100	90	106
Gender†
Male	44.6%	45.3%	46.5%	50.0%	37.5%	3.15
Female	55.4%	54.7%	53.5%	50.0%	62.5%
AIDS knowledge (Score range 0∼15)	9.15 (3.08)	10.29 (2.13)	9.43 (2.97)	8.66 (3.09)	8.20 (3.55)	6.74***	(1,3) (1,4) (2,4)	8.97 (2.95)	9.05 (3.23)	-0.20
Reproductive health skills (score range 4∼15)	9.36 (1.83)	9.30 (1.77)	9.52 (1.92)	9.27 (1.90)	9.34 (1.77)	0.29		9.26 (1.87)	9.49 (1.73)	-1.08
Self-efficacy (score range 1∼5)	2.34 (1.28)	2.29 (1.29)	2.55 (1.30)	2.42 (1.23)	2.11 (1.25)	1.85		2.52 (1.31)	2.31 (1.26)	1.36
Parent monitoring (score range 1∼5)	4.17 (0.93)	4.03 (0.93)	4.01 (1.04)	4.36 (0.76)	4.30 (0.91)	3.32*	(2,3)	3.92 (1.02)	4.40 (0.76)	4.50***
Expectations to engage in risk behaviors†
Smoke marijuana#	2.8%	4.0%	3.0%	1.1%	2.8%	-		4.7%	1.6%	-
Drink alcohol	15.9%	15.3%	21.4%	21.6%	6.7%	6.33		22.3%	12.6%	4.17*
Involve in pregnancy	10.7%	15.2%	10.1%	12.5%	5.7%	2.97		18.9%	3.8%	15.13***
Have sex	14.0%	18.6%	15.5%	17.2%	5.8%	8.44*		23.5%	6.0%	20.78***
Have protected sex if I have sex	39.9%	45.9%	42.9%	50.6%	21.6%	19.35***		41.7%	38.1%	0.43

Note: all test statistics (t, F, χ²) were adjusted using the variance inflation factors (VIFs);

^†Pearson's χ² test;

^#Fisher's Exact Test.

^*P<0.05;

^**P<0.01;

^***P<0.001.

Results

Implementation

Thirteen (76%) of the 17 eligible sixth-grade teachers initiated FOYC, among whom eleven had received training (five had attended the July workshop and six had received prior FOYC training). Five (29%) of the teachers fully implemented the 10 sessions of FOYC; three teachers had attended the July workshop, one had received prior FOYC training and the other had neither. All of the four teachers who did not implement FOYC had attended a previous FOYC training (but not the most recent July training). Two of the teachers who initiated and completed all ten sessions did not complete any of the measures and did not administer the student questionnaire to their students.

Implementation clustered by school. None of the four teachers at NPB initiated FOYC, while all of the teachers in the remaining schools initiated implementation. Among the eleven teachers who initiated the classes, all three teachers from NPD completed all 10 sessions; four teachers from NPA completed ≥4 sessions and four teachers (three from NPC and one from NPA) completed two.

Teacher assessments: Pre-and post implementation surveys and year-end interviews (data available from the authors)

Only five teachers completed the pre-implementation survey: All five of these teachers subsequently implemented at least the first two sessions of FOYC. Three teachers had attended a prior FOYC training workshop and had taught FOYC.

All five teachers reported that “prevention programs are very important for youth” and endorsed the statements that HIV prevention is “very important for youth in general,” “for grade 6 youth”, and for “the grade 6 youth in their communities.” Four teachers said that FOYC (as opposed to other HIV curriculum) “is very important for the grade 6 students in their school” while one said that it is somewhat important. Four teachers said that “the time spent teaching FOYC is about the same compared to the time spent teaching reading skills in grade 6” while one teacher (who went on to completed \4 sessions) said “teaching FOYC is less important.” Three of the teachers (all of whom completed only two sessions) felt that they would be very comfortable teaching FOYC, while the other two teachers, (one of whom had taught four sessions and the other had taught all 10 sessions), felt that they would be “somewhat comfortable”.

Nine teachers (all from NP) completed the post-implementation survey: three of these teachers had also completed the pre-implementation survey. Among the nine, two teachers completed two sessions, four teachers complete six to seven sessions and three teachers completed all 10 sessions. All endorsed the two statements that HIV prevention is “very important for youth” and for “the Bahamian community.” As well, all endorsed the importance of prevention programs in general for youth, while seven teachers said that HIV prevention programs are “very important for grade 6 youth in general” and for “grade 6 youth in their school”. The remaining two teachers responded that it is “somewhat important” for both categories; both of these teachers taught six to seven sessions of FOYC). Six teachers said that “the grade 6 students benefited “very much” from FOYC” (two of the three who completed all 10 sessions, three of the four who completed six to seven sessions and one of the two who completed two sessions) while the remaining three said students benefited “somewhat” from it. Seven teachers said that “the time spent teaching FOYC was “about the same” compared to the time spent teaching reading skills and two (one who taught four to six sessions and one who taught two sessions) said “teaching FOY was ”less important”.

Interviews were successfully conducted with 15 of the teachers, including the 11 who initiated implementation. All 11 of the implementing teachers expressed the view that the FOYC content was important. However, because the curricular material was not incorporated into the Bahamian national formal testing program, it received less emphasis in the curriculum than subjects which would be the focus of the national testing. All 11 expressed concerns that teaching FOYC required considerable preparation time compared to other subject areas. Teachers reported that preparation of the material for each session, which previously had been performed by a research project team, now resided with the teacher. Further, many of the teachers found it difficult to accommodate the presentation of the material within the allotted time frame). Three teachers expressed concern that the materials were too graphic for the youth. All of the teachers, regardless of level of FOYC training, expressed discomfort presenting the contraceptive information to grade six students. Two teachers felt that although the materials should be taught, the classroom teacher should not teach this material both because of their lack of expertise and because of the discomfort the students might feel.

The four teachers at NPB who did not implement FOYC expressed commitment to the program but encountered a variety of logistic issues including the need to replace a teacher who transferred in the fall.

Observer assessments and Teacher-self-assessment

Scheduling the observations was problematic. Despite appointments being made on several occasions, completion of two observed session for all 11 classes in which implementation occurred was not possible; ultimately one observation per class was completed for all classes. Since few teachers completed their teacher check-lists, a comprehensive comparison of teacher self-report with observations was not possible, although in general the teachers appeared to be reliably reporting what they had taught.

Student Data: Pre-and Post-intervention

Baseline and year-end questionnaires were administered to the students in all 15 classrooms in the four NP schools but were not administered in the two FI schools. Among the 15 classes completing the baseline questionnaires, there were 395 students (219 females and 176 males) with a median age of 11 years. The four schools did not differ with respect to gender distribution. HIV-related knowledge varied considerably by school with a high score of 10.29 at one school compared to a low of 8.20. As shown in Table 3, perceived parental monitoring was high overall (4.17 on a scale of 1 to 5), and reflected gender differences, with girls indicating significantly higher perceived monitoring. At baseline, the overwhelming majority of youth [(370(94%)] had not engaged in sex during the prior six months; males [17(10%)] were more likely than females [8(4%), p<.05] to report sexual activity. There was considerable variation in anticipated involvement in risk behaviors by school.

In Table 4, we examine the pre-post test data overall and according to number of classes completed. As shown in the first two columns, over-all, AIDS/HIV knowledge and protective sexual health skills increased significantly. Intentions to use protection if a youth were to have sex increased significantly while expectations to engage in sex did not change. Self-reported sexual activity within the last six months increased significantly.

Table 4. Knowledge, perceptions and behaviors regarding risk and protective factors among sixth-grade students in The Bahamas at baseline and six-months follow-up, by number of sessions of the Focus on Youth in the Caribbean (FOYC) curriculum implemented to their classes.

Variable	Overall		adj. t	No sessions (4 classes)		adj. t	2 sessions (2 classes)		adj. t	4-9 sessions (6 classes)		adj. t	All sessions (3 classes)		adj. t
	Base- line	Follow -up		Base- line	Follow -up		Base- line	Follow -up		Base- line	Follow -up		Base- line	Follow -up
Sample size	395	311		99	85		103	77		93	66		100	83
AIDS knowledge (Score range 0∼15)	9.15 (3.08)	10.48 (2.94)	4.90***	10.29 (2.13)	9.72 (2.96)	1.26	8.63 (2.97)	10.23 (2.95)	3.03**	8.16 (3.73)	11.33 (2.35)	5.48***	9.43 (2.97)	10.82 (3.16)	2.59**
Protective health skills (score range 0∼8)	4.89 (1.33)	5.14 (1.70)	1.97*	4.82 (1.12)	4.86 (1.77)	0.19	4.97 (1.46)	5.13 (1.57)	0.64	4.99 (1.37)	5.20 (1.59)	0.79	4.79 (1.38)	5.39 (1.84)	2.20*
Parent monitoring (score range 1∼5)	4.17 (0.93)	4.19 (0.79)	0.29	4.03 (0.93)	4.15 (0.78)	0.90	4.32 (0.78)	4.14 (0.87)	1.42	4.33 (0.91)	4.17 (0.82)	1.04	4.01 (1.04)	4.30 (0.70)	2.11*
Expectations to engage in risk behaviors†
Smoke marijuana	2.8%	0.7%	2.52	4.0%	2.4%	0.22	2.0%	0.0%	0.87	2.2%	0.0%	0.82	3.0%	0.0%	1.45
Drink alcohol	15.9%	22.1%	2.44	15.3%	13.1%	0.10	20.8%	22.4%	0.03	5.4%	25.8%	7.57**	21.4%	28.1%	0.61
Involve in pregnancy	10.7%	7.8%	0.98	15.2%	7.2%	1.59	13.9%	5.2%	2.06	3.2%	9.1%	1.42	10.1%	9.8%	0.01
Have sex	14.0%	16.1%	0.59	18.6%	19.5%	0.03	16.2%	18.2%	0.13	5.4%	12.7%	2.57	15.5%	13.4%	0.15
Have protected sex if I have sex	40.0%	66.7%	48.09***	45.9%	75.0%	15.40***	46.4%	66.7%	7.03**	22.4%	54.6%	16.63***	42.9%	68.3%	11.65***
	Overall			χ2		≤2 sessions			χ2		≥ 4 sessions			χ2
Engage in risk behaviors†	Base-line		Follow-up			Baseline		Follow-up			Baseline		Follow-up
Had sex in the last 6months	6.4%		10.6%	4.06*		6.5%		12.7%	3.86*		6.2%		8.5%	0.64
Ever had anal sex	2.5%		6.9%	7.68**		2.0%		6.3%	4.30*		3.1%		7.6%	3.53

Note: all test statistics (t, F, χ²) were adjusted using the variance inflation factors (VIFs);

^†Pearson's χ² test;

^*P<0.05;

^**P<0.01;

^***P<0.001.

Numbers in parentheses are standard deviations.

The four sets of columns to the right of the overall data display pre-post data by number of sessions of FOYC completed including “no sessions (four classes), two sessions (four classes), four to nine sessions (four classes) and all 10 sessions (three classes). These data reveal a pattern of increased knowledge, protective sexual health skills and perceived parental monitoring with increased numbers of sessions. Intention to use protection if engaging in sex increased significantly among all youth in all groups irrespective of receipt of classes. Reported expectation to consume alcohol increased significantly among youth who received four to nine sessions. Reported sexual activity in the last six months increased significantly among youth who received no, or only one or two FOYC sessions; reported sexual activity did not increase among youth who received four or more FOYC sessions.

Discussion

The roll-out of implementation of an effective HIV risk-prevention program across a nation offers a unique opportunity to both inform the science of implementation and to improve the implementation process of the national effort. The findings from the preparatory phase of national implementation in The Bahamas identified many logistic issues encountered by teachers as they attempted to implement a curriculum and employ measures that had been previously used but in a highly supported research settings and/or smaller more centralized efforts (CDC 2008; Deveaux et al. 2011). The overall low rate of full implementation (18%) is consistent with rates previously reported in the literature (Ringwalt et al. 2004).

Recommendations from the teachers and our observations have been used to modify subsequent training workshops and measures for future phases of national implementation. Moreover, the findings from this first phase that implementation is possible and can be effective in a real-world setting will be used to increase the likelihood of implementation in subsequent phases. In particular, the improvements in knowledge and skills and the suggestion of lower risk participation among youth who received more FOYC sessions should serve as encouragement to teachers in subsequent stages of national implementation to implement as much of the curriculum as possible.

The logistic issues began with workshop attendance. In contrast to the earlier effectiveness trial of FOYC in which all teachers participated in the workshops, in this first phase of national implementation, only seven of the 17 teachers who were asked to implement FOYC attended. Likewise, in contrast to the effectiveness trial of FOYC in which all sessions of the curriculum were delivered to all classes, six of the teachers never began implementation and only three teachers completed all 10 sessions. The teachers discussed the many implementation issues, some of which (starting too late in the year) could be easily addressed in the future. Others will require intervention from school administrators or the MOE (class cancellations, changing teachers, and/or including questions from the FOYC curriculum in the Bahamian national formal testing program) in subsequent phases. Although with reminders and prompting from the research staff, many of the teachers had completed the self-checklist in the FOYC effectiveness trial, in this real-life setting, few teachers actually completed it, commenting that the format was cumbersome. As a result, we have simplified the measure employing a more user-friendly format. Finally, in response to the legitimate concerns about time required to prepare materials for teaching FOYC, the MOE has altered the manuals to reduce this time commitment.

Somewhat surprising, given the extant literature (Green et al. 2009; CDC 2008; Hallfors and Godette 2002) and our own experience with FOYC, was the lack of correlation between teacher training workshops and delivery of FOYC. However, in view of the evidence from some studies of the importance of detailed teacher training manuals (Dusenbury et al. 2003), it is possible that since all of the teachers did receive a detailed teacher manual the lack of exposure to a training workshop was less important. A contrary explanation is suggested by the experience of Kelly and colleagues (2000) who found that the training workshop was less important in implementation than ability to confer with skilled facilitators after the workshop during field implementation. This phenomenon may explain the strong correlations between performances within a school; for example, the six teachers who did not initiate implementation represented 100% of the teachers at two schools and no other teacher failed to initiate implementation. The three teachers who implemented all sessions were in one school and the teachers in the remaining three schools all implemented between some but not all sessions. It may be that the teachers at “high implementing schools” used each other as encouraging think-tanks to discuss problematic issues as they arose. These findings underscore the importance of encouraging teachers whether or not they are able to attend the training workshop to support each other and to share materials and experiences in their efforts to implement FOYC. This advice has been written in the training manual which all teachers will receive whether or not they attend the workshop. The clustering of implementation by school may also reflect the previously described positive correlation between implementation and supportive local authorities (Dusenbury et al. 2003).

The positive correlation between number of sessions of FOYC completed and changes in several key outcome indicators is encouraging. This finding is consistent with the finding in the Blakely study (Blakely et al. 1987), indicating a positive correlation between implementation and outcome. Given the low rate overall of intervention implementation (only three teachers among 17), this direct correlation between delivery and outcome offers an important opportunity for future training workshops. We anticipate that if teachers are confident that their efforts will have an impact, they may be more likely to overcome logistic hurdles impeding delivery of FOYC in the classroom. The observation that risk behaviors increased between the pre- and post implementation surveys is consistent with the well documented increase in risk-taking behavior during adolescence (Galbraith et al. 2009).

Potential limitations

This is the first phase of national implementation; in contrast to an effectiveness trial, each phase of national implementation will be used to influence (and improve) subsequent phases. Because of the relatively low number of self-checklists returned and observations completed for the same session, we are not able to confirm the accuracy of the self reports or to comment as to whether the exercises that were completed were taught as intended by the developers.

Implications

Conclusions

The data obtained from this first phase of national implementation is being used by the Ministry of Education to inform the roll-out of subsequent implementation phases in all grade six classes throughout the Bahamas. Particularly important findings from this study are the cohort (school) effect of implementation and the strong correlation between amount of curriculum implemented and behavioral outcomes. Although the measures had been employed in a study setting previously, application in a “real life” setting in the pilot study revealed many issues which had not previously been problematic and required addressing. Prior research regarding teacher workshops and subsequent implementation success has underscored the need for more than content and exposure if teachers are expected to follow through on implementation (Fixsen et al. 2005; Deveaux et al. 2007; Joyce 1992). Given the findings from the first phase of implementation, structured work on the comfort level of teacher with the subject matter is needed in the training. In addition, efforts will be made by the MOE to afford more time during training to discussing teacher concerns about the curriculum and to provide more assistance to teachers throughout the year as they teach the curriculum. These findings underscore the unique opportunities offered by a national implementation effort to both implementation science and to the national effort itself.