Using implementation strategies to promote the maintenance of Kids SIPsmartER in rural Appalachian middle schools: a process evaluation

Abstract

Background

As few evidence-based behavioral interventions (EBBIs) are sustained in schools, their population health impact is limited. Incorporating implementation strategies can facilitate their sustainment. The Kids SIPsmartER intervention, which incorporates a robust implementation strategy bundle, has been implemented in 12 rural Appalachian middle schools since 2018. The trial design supported its maintenance after the initial Researcher-Led Year allowing for continued collection of implementation data.

Methods

This concurrent mixed method process evaluation included six data sources: fidelity checklists, teacher interviews and surveys, principal interviews and surveys, and researcher-maintained field notes. Included data came from each school’s first and/or second Teacher-Led Years (i.e., after the Researcher-Led Year). Quantitative data were summarized using descriptive statistics. Summaries of schools’ implementation experiences were developed from qualitative data sources and then content coded.

Results

Data from 24 teachers and 11 principals from all 12 schools were included. Of the 10 schools that completed their Researcher-Led year by Spring 2022, 9 (95%) maintained Kids SIPsmartER. 95% of teachers attended formal pre-implementation meetings. Teachers rated the various implementation strategies’ helpfulness from 4.2 to 4.8 [5-point scale]. Teachers delivered 97% of expected lessons. Average fidelity among returned checklists was 94% (SD = 5.0%), which decreased to 66% (SD = 40%) when missing forms were counted as zeros. All continuing schools modified the delivery timeline. Some modified lessons. Rationales for delivery decisions reflected considerations to increase program fit. Multiple factors influenced teachers’ and principals’ perceptions of Kids SIPsmartER’s fit, including compatibility with the larger curriculum, filling an instructional gap, and potential to impact students. Teachers and principals rated lesson content as important: 4.61 and 4.38 [5-point scale], respectively. Teachers also strongly agreed that their students engaged with the intervention [6.33, 7-point scale].

Conclusions

Study findings highlight the importance of incorporating and evaluating implementation strategies to ensure intervention fidelity and foster teacher motivation. Also, they stress the necessity of ensuring implementation strategies reduce school-level barriers and foster fidelity-consistent adaptations, teacher motivation, and intervention scalability. Together, findings have implications for the translation of Kids SIPsmartER and school-based EBBIs into practice, which could impact population health.

Trial registration

Clincialtrials.gov: NCT03740113. Registered 14 November 2018 – Retrospectively registered. https://clinicaltrials.gov/study/NCT03740113.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12889-025-23554-x.

Background

Energy-balance related behaviors (i.e., nutrition and physical activity behaviors) have long been associated with risk for multiple preventable chronic diseases, including obesity, diabetes, and cancer [1, 2]. Establishing healthy energy-balance related behaviors during childhood and adolescence has been recognized as a key means to reduce risk and promote population health [3]. Since approximately 95% of children and adolescents attend public or private schools in the United States [4], schools have long been promoted as a population health strategy to deliver interventions to encourage engagement in these behaviors [5–7]. While there are existing evidence-based behavioral interventions (EBBIs) designed for schools that target energy-balance related behaviors [8], few have been maintained and even fewer sustained after the research period [7, 9]. This limits the potential population health impacts of these interventions.

To address this gap and promote population health, the barriers that hinder the implementation of these EBBIs need to be overcome. A common barrier is the actual and perceived fit of EBBIs within the school, as EBBIs often do not address tested standards of learning, are more intensive and longer than typical health units, and have strict implementation protocols [7, 9, 10]. Other barriers include balancing the EBBI with other priorities, teacher time to deliver and prepare, cost/funding, and access to training and technical assistance [7, 9, 10].

Implementation strategies can help promote the adoption, implementation fidelity, maintenance, and sustainment of EBBIs and reduce barriers to their implementation [11]. They can be implemented as discrete strategies or “bundled” packages. Yet, research regarding execution of implementation strategy approaches within the context of schools is far less advanced than in healthcare settings [12, 13]. Fortunately, there has been growing momentum to advance implementation science in schools. Notably, the implementation strategies from the Expert Recommendations for Implementing Change (ERIC) Project [14], which largely focused on healthcare settings, has been translated to the school setting. The School Implementation Strategies, Translating ERIC Resources (SISTER) is a comprehensive school-specific compilation of implementation strategies that includes 75 specific implementation strategies organized across nine categories [12, 13]. By providing a common nomenclature, SISTER allows researchers and interventionists a means to clearly and systematically describe the implementation strategies used to execute and evaluate school-based EBBI. This can increase the evidence of what activities most readily facilitate faithful implementation and how these approaches are executed.

Kids SIPsmartER is a 6-month multi-component behavioral intervention designed to decrease sugar-sweetened beverages (SSB) among middle school students and their caregivers [15]. It is guided by the Theory of Planned Behavior [16], health and media literacy concepts [17, 18] and the RE-AIM (reach, effectiveness, adoption, implementation, and maintenance) framework [19]. Kids SIPsmartER consists of a 12-lesson classroom-based intervention, which is implemented in 7th grade physical education (PE) and/or health class. Classrooms lessons include content related to health risks of consuming too many SSBs and how marketing can influence SSB intake. Lesson activities were aimed at helping students develop and follow through with personalized action plans and to support them to create public service announcements (PSAs) related to drinking fewer SSBs. How curricular content and activities align with Kids SIPsmartER’s conceptual foundations is described in detail elsewhere [15]. Additionally, Kids SIPsmartER includes a short message service (SMS) intervention for caregivers that provides educational and strategy messages for enrolled caregivers [20].

Kids SIPsmartER was tested for effectiveness through a type 1 hybrid effectiveness-implementation and cluster randomized control trial that started in 2018 (Clincialtrials.gov: NCT03740113; 2018–2022) [15, 21]. Findings demonstrate that Kids SIPsmartER has facilitated statistically and clinically significant reductions in SSB intake among students, particularly among high SSB consumers at baseline, when delivered by researchers [21] and by teachers [22]. Furthermore, research has shown that Kids SIPsmartER is acceptable to students and their caregivers [23]. Together, these findings suggest Kids SIPsmartER is a promising intervention.

Furthermore, the design of the Kids SIPsmartER trial allowed all schools to receive Kids SIPsmartER regardless of randomized assignment and promoted maintenance of the intervention after the initial year of delivery. The intervention and type 1 hybrid effectiveness-implementation trial are described in more detail in the Methods and elsewhere [15]. However, in brief, primary outcome data (i.e., student SSB intake) were collected during an initial Researcher-Led Year. Then, implementation and secondary outcomes were collected during subsequent Teacher-Led years. Thus, Kids SIPsmartER’s design included a robust exploration of its implementation after the initial researcher driven research period. This created an opportunity to gather evidence from this promising intervention to fill the literature gap related to understanding how EBBIs, particularly those targeting energy-balance related behaviors, can be sustained in schools. Specifically, this mixed-methods study sought to answer the following four research questions related to the implementation dimension of RE-AIM: (i) How did teachers and principals engage with and perceive Kids SIPsmartER’s implementation bundle activities?, (ii) What was teacher fidelity to the implementation protocols?, (iii) How did teachers and principals approach delivering Kids SIPsmartER and what informed their decisions?, and (iv) How did teachers rate and describe the fit of Kid SIPsmartER within their school?.

Methods

This study was a concurrent mixed-methods process evaluation [24] of the implementation of the classroom curriculum component of the Kids SIPsmartER trial in 12 rural Appalachian middle schools. Study procedures were approved by the University of Virginia Institutional Review Board for the Social and Behavioral Sciences. Informed consent for teachers and principals was waived by this body, as consent was implied by their agreeing to participate in the study’s implementation activities, which included completing the interviews.

Kids SIPsmartER implementation bundle

To support the implementation of Kids SIPsmartER, the research team developed an implementation strategy bundle that incorporated multiple strategies identified in SISTER [12, 13]. These strategies were operationalized through implementation activities. When developing the implementation bundle, a key intention was to ensure schools had the necessary support to faithfully implement Kids SIPsmartER while allowing relative flexibility to make the program “fit” within their classrooms.

Kids SIPsmartER’s implementation activities and their associated SISTER implementation strategy categories are detailed in Table 1. Included activities reflect all nine of the SISTER implementation strategy categories. In all years, the Kids SIPsmartER implementation bundle included activities related to manualized curriculum, teacher training and technical assistance, fidelity checklists, and a website that contained teacher training resources and student curricular materials. During each school’s first year of implementation, Kids SIPsmartER was predominantly delivered by researchers and with teachers supporting delivery (i.e., Researcher-Led). Teacher support ranged from helping with classroom management to co-teaching parts of the lessons. During this Researcher-Led Year, nine core lessons were delivered weekly in the fall, and the other three lessons were delivered as “booster lessons” approximately every two weeks in the spring.

Table 1.

Mapping of Kids SIPsmartER’s implementation bundle activities to SISTER implementation strategy categories

Kids SIPsmartER Implementation Protocol Activity		SISTER Implementation Strategy Category
		1	2	3	4	5	6	7	8	9
1	Detailed implementation plan with flexibility built in for teachers to adapt mode, timing, and scope of delivery (after Researcher-Led Year)	✓		✓
2	Yearly in-person teacher onboarding, including training and planning activities	✓			✓	✓	✓
3	Presentation of key implementation, process, and outcome findings on an ongoing basis to teachers and school leadership	✓			✓		✓
4	Post-implementation interviews with teachers (yearly)	✓			✓		✓
5	Post-implementation interviews with principals (yearly)	✓					✓
6	Fidelity sheets completed by teachers after each lesson	✓
7	Lesson observations conducted by research staff for three lessons per school	✓
8	Student-level summative evaluation	✓
9	Caregiver-level summative evaluation	✓
10	Informal remote support provided to teachers upon request or if not delivering as planned		✓			✓
11	Lesson overview videos (accessible through website)		✓
12	Researchers deliver classroom curriculum with teacher support during school’s first year of implementation (Research-Led Year)				✓	✓
13	During school recruitment, program content and structure framed in the context of regional health behaviors and access to nutrition-related primary prevention programs.				✓
14	Emailed communications to school leadership throughout program implementation				✓
15	Optional within and between district networking via annual in-person teacher training				✓
16	Formal approval for program from district- and school-level leadership required				✓					✓
17	Manualized curriculum and implementation guide, including lesson plans, lesson-relevant background information, notes for PowerPoint slides, activity how-to sheets, and teacher how-to videos					✓
18	Manualized curriculum and implementation guide distributed as hard copies and available electronically					✓
19	Multiple teacher training and technical assistance methods (e.g., in-person training, online teacher training videos, and informal support) available throughout program implementation					✓
20	Email reminders sent to teachers about upcoming activities and related training and curricular resources						✓
21	Option to have teams deliver the curriculum that may include staff outside of the health/PE teachers (e.g., nurse)						✓
22	Curricular materials aligned to SOLs							✓
23	Program pitch opportunities to students and parents (e.g., tabling at back-to-school nights, in class presentations)							✓
24	Monetary compensation for teachers’ out of school time								✓
25	School-level stipends								✓
26	Curricular materials replaced other health education activities (as appropriate)									✓
27	Options to alter health/PE environment to accommodate curriculum (e.g., create teaching space outside of gym, teach all students together)									✓

1 = Use evaluative and iterative strategies 2 = Provide interactive assistance

3 = Adapt and tailor to context 4 = Develop stakeholder interrelationships

5 = Train and educate stakeholders 6 = Support educators

7 = Engage consumers 8 = Use financial strategies

9 = Change infrastructure

✓ activity maps on to one or more specific strategies within each category

Participating schools

Twelve middle schools located in four rural counties in Appalachian Virginia and West Virginia participated in the Kids SIPsmartER trial [25]. These schools are situated in a region that experiences notable disparities related to engagement in unhealthy energy-balance behaviors (i.e., high rates of SSB intake), rates of preventative health conditions, and supportive social determinants of health [26–28].

Measures and data collection

This manuscript includes implementation-related data from six sources: teacher-reported fidelity checklists, teacher interviews, principal interviews, teacher surveys, principal surveys, and research staff field notes. The fidelity checklists, interview protocols, and surveys were specifically developed for this study (See Supplementary Material 1). Together, these sources assessed indicators of engagement with, perceptions of, and adaptations to the implementation activities; intervention fidelity; descriptions and rationales for delivery approach; and perceptions of the program’s fit. The process data included in this manuscript were collected over the 2018–2019 to the 2022–2023 school years (SY).

Figure 1 illustrates the study design, schools’ involvement, and included data collected each year. Due to its cluster randomized design [15], schools started to deliver Kids SIPsmartER during different years starting with SY 2018–2019. Because of this, schools had completed between zero and four Teacher-Led Years by the end of SY 2022–2023. Therefore, to allow for data to be more evenly distributed across schools, only relevant data sources from a school’s Researcher-Led Year (i.e., first year of delivery) and first two Teacher-Led years (i.e., second and third years of involvement) were included. Also, due to the unique impacts to classroom teaching caused by COVID-19 restrictions during SY 2020–2021, data related to fidelity and teaching approaches for this SY were not incorporated into this manuscript. However, a description of the delivery of Kids SIPsmartER during this year is available elsewhere [29].

Fig. 1.

Kids SIPsmartER trial design and school engagement per year

Fidelity checklists

Teachers were expected to complete lesson-specific checklists after each delivered lesson. These checklists collected data about when a lesson was delivered, the use of specific delivery-focused implementation activities (e.g., lesson plans, lesson videos), completeness of the delivery of key lesson components (0 = not completed, 0.5 = partially completed, 1 = completed), and mode of delivery (in-person, virtual, hybrid). Modifications were captured with open-ended response options. The checklists also measured teachers’ perceptions of student engagement and their self-efficacy in delivering the lesson on 7-point agreement scales (1 = strongly disagree to 7 = strongly agree). Teachers could complete fidelity checklists either via either a hardcopy form or electronic link.

Field notes

Research team members recorded field notes after in-person, phone, and email-based interactions with teachers. The field notes captured when teachers delivered lessons, approaches teachers took to deliver lessons, and teacher concerns and barriers.

Teacher and principal interviews

The approximately 30-minute audiotaped interviews were conducted by one of two trained research staff. While these staff had been actively involved in the design and implementation of the Kids SIPsmartER classroom curriculum, neither were involved with the delivery of Kids SIPsmartER at the schools for which they conducted interviews. The semi-structured interview protocols sought to understand schools’ specific experiences delivering Kids SIPsmartER, including decisions about delivery, barriers to delivery, perceived benefits, and program fit. They also included questions about engagement with and perceptions of aspects of the implementation bundle. Questions were differentiated between teachers and principals.

Teacher interviews were conducted up to twice a year (mid- and end-of-year) during Researcher-Led and Teacher-Led Years. After SY 2020–2021, the number of interviews during the Teacher-Led Years was reduced to one, and only during the first Teacher-Led Year. This change occurred to reduce teacher burden, as the multiple and repeated interviews were not eliciting new information. Principal interviews were planned to be conducted in the spring of a school’s Researcher-Led Year. However, interviews with principals from the four schools who ended their Researcher-Led Year in spring 2020 were conducted at the end of their first Teacher-Led Year, due to the COVID pandemic causing school shutdowns in March 2020.

Teacher and principal surveys

At the middle and end of their Researcher-Led Year, teachers completed a short survey in which they rated their perceptions of the importance of the curriculum content (1 = not important, 5 = very important; 14-items) and the usefulness of the specific implementation activities (1 = not useful, 5 = very useful; 6-items).

The principal’s survey was administered verbally during the interview. Principals were asked to rate the importance of aspects of the curriculum’s content (1 = not important, 5 = very important; 14 items).

Data analysis

Quantitative data were analyzed using SPSS version 29 (Armonk, NY). Interviews were transcribed using the Trint transcription service [30]. Researchers reviewed and cleaned transcripts before analysis. Related qualitative and quantitative data were triangulated during analysis.

Intervention maintenance

The proportion of schools that maintained Kids SIPsmartER was calculated as the number of schools that completed the first Researcher-Led Year divided by the number that had completed their Researcher-Led Year.

Implementation bundle activities

Using data from completed fidelity forms, the use of six specific implementation activities (i.e., lesson plans, student facing slides, other manualized curriculum components, teacher-facing videos, student-facing videos, informal meetings with research staff) was calculated as the proportion of lessons for which teachers reported using each activity to prepare for lesson delivery. Proportions were calculated for the first and second Teacher-Led Years individually and together. Average teacher ratings of the usefulness of five implementation activities collected from the teacher surveys were calculated.

Fidelity

Five indicators of fidelity were assessed using data from the teacher fidelity checklists, researcher field notes, and teacher interviews. Four of these indicators were from quantitative sources: proportion of expected lessons delivered by teacher, proportion of fidelity checklists returned by teacher, overall lesson fidelity based on completed fidelity checklists, and overall fidelity based on expected fidelity checklists (i.e., checklists for lessons that were expected to have been completed but were missing were scored as a zero). Reported modifications were analyzed through conventional content coding [31] using the process described in Delivery Approaches.

Delivery approaches

We content coded qualitative data from teacher and principal interviews and field notes collected during the Researcher-Led and Teacher-Led Years related to planned and executed delivery approaches during the first and second Teacher-Led Years. During this coding process, we also content coded modifications and perceived barriers and benefits to the program. We used a multi-step coding process that involved the creation of structured summaries. The decision to use structured summaries was driven by the need to see the “big picture” of a school’s delivery decisions and experience across data sources. Sections of the structured summaries included timing and frequency of lessons, number of lessons taught, rationales for delivery decisions, modifications to lessons, barriers to delivery, and perceived benefits of Kids SIPsmartER. Summaries contained both synthesized data and quotes. First, pairs of researchers independently created structured summaries synthesizing relevant data from interview transcripts and field notes collected during Researcher-Led and Teacher-Led Years. Then, the researchers independently created an overall summary for each school that combined data from all individual yearly structured summaries. Second, a third coder merged the overall summaries for each school. During this merging process, this third coder brought together the data from the two summaries into a new document. Content that was different was highlighted. The third coder would review the documents to see if they could settle the difference. If they could not, they met with the original coders to discuss. This process allowed for the maintenance of a documented audit trail that demonstrated approximately 90% coder agreement [32]. Third, the overall summaries for each school were compiled into a single document to allow for synthesis of data across schools. Fourth, using conventional content coding [31], researchers independently synthesized descriptions of frequency and timing of lesson delivery, rationales for delivery decisions, descriptions of modifications to Kids SIPsmartER delivery both used and planned but not acted on, and benefits and barriers to Kids SIPsmartER across all 12 schools. Fifth, a third coder merged the independently developed cross-school delivery summaries using the same process as described above.

Kids SIPsmartER’s fit

Indicators related to the fit of Kids SIPsmartER in the school were collected through the teacher and principal interviews and surveys and fidelity checklists. Means were calculated for teacher and principal ratings of the importance of lesson content and activities; teacher ratings of student engagement; and teacher self-efficacy delivering the lessons. Teacher and principal interviews were inductively coded to identify factors (i.e., barriers and benefits) that impacted teacher and principal perceptions of the program as part of the coding process described in the preceding section.

Results

Table 2 presents a summary of teacher and principal engagement as well as the data collected from them each year. Quantitative data are summarized in Table 3. Qualitative findings are presented in Table 4. The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Table 2.

Summary of schools, participants, and data collection across Researcher-Led and Teacher-Led years

	Researcher-Led Year	1st Teacher-Led Year	2nd Teacher-Led Year
Schools
# delivering lessons	12	9	7
# did not return from previous year	n/a	1	0
Teachers
# delivering lessons	19	15	13
# new teachers	n/a	2	3
# interviews completed / expected	35/38	22/24	8/8
# surveys completed / expected	12/19	n/a	n/a
Principals
# principals	12	9	7
# new principals	n/a	1	0
# interviews completed / expected	8/12a	4/4	n/a
# surveys completed / expected	8/12a	4/4	n/a

^a 4 of the schools completed their Researcher-Led Year in the Spring of 2020. Three principals were unable to complete the interviews and surveys before the shutdowns. While expected per the protocol, we did not pursue their completing them due to the other burdens they were facing

^b We amended the protocol during the 2020–2021 SY to allow the principals from the four schools that completed their Researcher-Led Year in the Spring of 2020 to take part in the interview at the end of their first Teacher-Led Year

Table 3.

Quantitative implementation findings related to perceptions of implementation bundle activities and fidelity

Item	Teacher-Led Years Combined Mean (SD) or N (%)	1st Teacher-Led Year	2nd Teacher-Led Year
Implementation bundle activities
Data source: Field notes	n = 18 teachers	n = 15 teachers	n = 13 teachers
Teachers who took part in Researcher-Led Year	10 (56%)	10 (67%)	6 (46%)
Teachers who completed pre-delivery training	17 (95%)	15 (100%)	12 (93%)
Data source: Completed fidelity checklists	n = 11 teachersa, b	n = 8 teachersa	n = 7 teachersa, b
Proportion of lessons teachers used _____ to prepare for
Lesson plans	0.85 (0.29)	0.96 (0.08)	0.70 (0.40)
Student facing slides	0.76 (0.40)	0.98 (0.47)	0.47 (0.50)
Other manualized curriculum resources (i.e., activity resources, educator backgrounds)	0.61 (0.46)	0.72 (0.41)	0.46 (0.51)
Teacher facing lesson videos	0.56 (0.46)	0.67 (0.46)	0.43 (0.45)
Informal technical assistance meetings with research staff	0.09 (0.20)	0.14 (0.27)	0.02 (0.06)
Fidelity
Data source: Fidelity checklists and field notes	n = 14 teachersa	n = 8 teachersa	n = 10 teachersa
Proportion of expected lessons delivered	0.97 (0.10)	1.00 (0.00)	0.93 (0.14)
Proportion of fidelity checklists returned by teacher	0.72 (0.40)	0.90 (0.21)	0.58 (0.47)
Overall lesson fidelity, based on completed forms	0.94 (0.05)	0.94 (0.04)	0.94 (0.08)
Overall lesson fidelity, based on expected forms	0.66 (0.40)	0.85 (0.20)	0.47 (0.46)

^a Data collected from fidelity checklists completed by teachers during their schools’ first or second Teacher-Led Years that occurred during the 2020–2021 school year is not included

^b Of the 14 unique teachers who delivered Kids SIPsmartER during their school’s first or second Teacher-Led Year that did not occur during the 2020–2021 school year, three did not complete any fidelity checklists

Table 4.

Factors that influence the delivery and reception of Kids SIPsmartER

Concept	Exemplar Quotes
Delivery approaches
Timing delivery to match with other needs	We would probably pick two days a week that we could do the content during those seven weeks in the middle [of the nine week health cycle] … that gives us an opportunity to have to make ups if we needed to. (Teacher, School 10) When I go to teach the program with seventh graders next year, I would I think what I would do is I would try to break it down. And then do more lessons a week to try to make it flow with the content that I’m teaching in other areas. Just to make it more of an everyday thing versus just once week or twice a week. (Teacher, School 11)
• To maximize time spent in physical education
• To fit with existing formal or informal health schedules
Selecting class location and size to enhance teaching/learning experience	I think it would go a lot better in a classroom setting with 25 kids. (Teacher, School 9)
• Space to deliver the lessons that is conducive to learning
• Supporting class sizes that foster a better learning environment and can hold number of students
Making adjustments to Kids SIPsmartER content or structure to increase fit	I think every lesson, you know, I threw something in there, rather it was a personal experience, you know, or just or something that I knew also about the situation. (Teacher, School 3)
• Adding personal anecdotesa
• Adding more hands-on and/or visual activitiesa
• Changing the number of teach lessons a
• Extending or shortening lesson time
• Re-organizing content
• Spreading content over multiple grades
Kids SIPsmartER’s Fit
Positive factors
Kids SIPsmartER fills a need related to accessible, modern, and well-designed health education curricula	We don’t have a health curriculum here, so … it’s helping us or giving us lessons to teach because I mean otherwise, I just got to find something. … But this is something that will help us. (Teacher, School 2) We would use this going forward just because it is so nice to be able add it to our curriculum and add it to our lessons. … Having that set structure and those lessons already developed, is going to be a huge moving forward. (Teacher, School 5)
Content and activities are appropriate for 7th graders	I think it’s an essential program that kids need. They need this knowledge and information before it’s too late because they’re still at that age where their minds can be kind… they’re maturing, their body is changing. So I think it’s important to get this knowledge to them so they are well aware and they have the resources to make changes, even if they’re not huge changes, little changes over time will make a difference. (Teacher, School 6)
The included hands-on and audio-visual based activities promote engagement.	It’s a good modern, you know, I guess, health curriculum that, you know, catches their attention. (Teacher, School 4)
Teachers feel well-supported by the Kids SIPsmartER team and materials to implement the program.	I think it’s really set up well step by step. And everything is easy to read, … You’ve [also] got the resources on the Internet. The lesson is all spelled out in the binder. (Teacher, School 1)
Teachers and principals observe changes in students’ awareness and behaviors related to sugar-sweetened beverages	I think it’s helping, at least if they’re not changing, at least they’re thinking about it. (Teacher, School 2) They act like they hate it and stuff. But you can see them … you hear conversations here and there about how, you know, “Monster - I don’t drink it anymore” or “I’ve kind of cut back on my PowerAde.” And … you see more of the kids bring in the Gatorade Zero. (Teacher, School 4)
Excessive sugar-sweetened beverage intake and related health conditions are regional public health concerns	I think it really opens these kids eyes, who just go home and pick up a soda, and they just think it’s hidden sugar. They don’t realize that it’s in there. And I think it’s really it’s really hit home with a lot of them. (Teacher, School 1)
Mixed factors
Kids SIPsmartER addresses expected standards of learning and/or health content, but there are other topics that need to be covered.	Even though it’s coming from Virginia, it coincides with our standards in West Virginia as well. So, I mean, it’s not like we’re having to circle the wagons or recreate the wheel here. I mean, the standards are driven. It aligns with what we’re doing anyway. (Teacher, School 11)
The design of Kids SIPsmartER’s lessons works best when delivered to traditional class sizes and in settings more aligned with normal classrooms.	Whenever we have smaller class sizes seeing it, the things that were a little harder to go over [were not as much of a problem]. So, while it may have been a barrier this year, it actually is kind of eye opening and helpful when it comes to planning for the future. (Teacher, School 11)
Negative factors
Students did not enjoy activities that felt overly didactic or removed from their daily life.	I know some of the kids have said, you know, like what’s you know, “How’s this going to help me now?” And it’s just kind of like they don’t, I guess, you know, as middle schoolers, they don’t understand the impact about things that are happening right now and then in the future. (Teacher, School 2)
Students perceived Kids SIPsmartER as taking away from gym time	Some of these kids, they live for gym class … It’s just they automatically have that negative attitude, because they’re getting pulled out of it. And, I know it’s only one day a week, but it, you know, like it builds because in PE we’ll do tournaments and they’ll have [inaudible] time of that. And so it kind of builds a resentment with them. (Teacher, School 10)

^a Adaptations that were both discussed as potential changes and were reported as being executed by teachers

Participants

Study participants included 24 teachers and 11 principals involved in the delivery of Kids SIPsmartER between SYs 2018–2019 and 2022–2023. All teachers except one were PE and/or health teachers; the other teacher was the school nurse. The teachers and principals were 83% (n = 14) and 55% (n = 6) male, respectively.

Of the expected 70 teacher interviews, 65 were completed (94%). All of the 24 participating teachers completed at least one interview. Twelve of the 19 teachers expected to complete surveys did so (63%). Eleven of the 12 potential principals (92%) completed surveys and interviews.

School-level intervention maintenance

Of the 12 schools engaged in the Kids SIPsmartER trial, one (8%) dropped from the study after its initial Researcher-Led Year. Of the 10 schools that had completed their Researcher-Led Year before SY 2021–2022, 9 (90%) maintained Kids SIPsmartER into the first Teacher-Led Year. All schools who progressed to their first Teacher-Led Year continued to the second Teacher-Led year. As illustrated in Fig. 1, there were 16 first and second Teacher-Led delivery cycles. Of these, 11 occurred outside of SY 2020–2021.

Implementation bundle activities

Among teachers delivering Kids SIPsmartER during their school’s first and/or second Teacher-Lead Year, 10 teachers (56%) participated in their schools’ Researcher-Led Year and 8 did not. Of those that did not participate in their schools’ Researcher-Led Year, five were new to their school during Teacher-Led Years. The three other teachers were from schools that could not have a Researcher-Led Year, as their schools’ planned year occurred in SY 2020–2021 during which there was highly limited access to the schools and schools were in a constant flux between in-person and virtual teaching.

Of the 18 unique teachers who delivered Kids SIPsmartER during their school’s Teacher-Led Years (including SY 2020–2021), 17 (95%) completed formal pre-implementation training before each year. The teacher who did not complete the training had not yet been hired by the school at the time of the training and turned down an offer for one-on-one training.

Teachers who delivered Kids SIPsmartER during their school’s first and/or second Teacher-Led Years that did not occur during SY 2020–2021 and had completed fidelity checklists (n = 11) reported using the six assessed implementation bundle activities to prepare for lessons at different rates. Lesson plans, which were part of the manualized curriculum, and student-facing slides were reported to be used for lessons, with overall rates of 85% and 76%, respectively. The other manualized curriculum resources (e.g., activity resources, educator backgrounds) and teacher-facing lesson videos were used to prepare for 61% and 56% of lessons, respectively. Informal meetings with research staff were reported to be used to prepare for 9% of lessons. Rates varied between the first and second Teacher-Led Years, with rates generally higher during the first year.

Overall, teachers rated the implementation bundle activities as useful to very useful in aiding them to prepare for lesson delivery. (Data not in table.) Lesson plans were the most highly rated [4.77 (SD = 0.44)] followed by lesson overview videos [4.54 (SD = 0.66)], in-person meetings [4.50 (SD = 0.54)], observing researchers deliver the lessons [4.23 (SD = 4.23)], and other aspects of the manualized curriculum [4.17 (SD = 1.11)].

Fidelity

Among the nine schools that delivered Kids SIPsmartER during their first and/or second Teacher-Led Years that did not occur during SY 2020–2021, 14 teachers were expected to return fidelity checklists. Overall, fidelity checklists were returned at a rate of 72%, with rates of 90% and 58% during the first and second Teacher-Led Years, respectively. Lesson fidelity scores ranged from 94% based on completed forms to 66% when including unreturned forms. Fidelity scores of completed forms did not vary by Teacher-Led year, but scores when unreturned forms were added were lower for the second year compared to first year (47–85%).

Teachers reported making modifications to lessons. Most frequently, they reported changing or adding examples (i.e., teacher using their own experience). The two teachers from School 3 reported consistently delivering lessons over two consecutive days, while the two teachers from School 6 reported consistently shortening lessons to allow for both the lesson and a physical activity to occur in the same class period. The teacher from School 8 decreased the number of expected lessons from 12 to five and only delivered the lessons virtually as part of activities for snow/inclement weather days.

Delivery approaches

In the 11 first or second Teacher-Led delivery cycles that occurred in SYs 2019–2020, 2021–2022, and 2022–2023, schools made adaptations to the number of lessons taught, timing of lessons, and mode of delivery. In ten of these delivery cycles (91%), schools planned to teach all 12 lessons; one school planned to deliver only five lessons. In all of these delivery cycles (100%), schools adapted the delivery timeline used in the Researcher-Led Year (i.e., 9 weekly lessons in the fall followed by 3 lessons delivered 2 weeks apart in the winter) and the delivery timeline was reduced. The most common approach was to deliver Kids SIPsmartER from the late fall to early spring when it was too cold or too wet to have students outside. This approach was used in eight (73%) of the delivery cycles. In ten cycles (91%), multiple Kids SIPsmartER lessons were delivered per week; only one delivery cycle maintained a one lesson per week schedule. Lessons were taught in one of three ways. In five cycles (45%), teachers maintained the delivery approach used in the Researcher-Led Year: in-person using the PowerPoint slides and workbooks. In the other cycles, teachers either taught in-person with the workbooks but using a combination of the PowerPoint slides and student facing videos (5, 45%) or taught asynchronously on a virtual platform (1, 9%).

Teachers’ and principals’ decisions on how to deliver Kids SIPsmartER reflected three factors: (i) ensuring Kids SIPsmartER’s delivery timing allows for needs and wants to be met, (ii) how class location and size would impact teaching/learning experience, and (iii) increasing fit of Kids SIPsmartER to student learning needs. Regarding the timing of lesson delivery, teachers and principals discussed how they wanted to maximize PE time, especially time spent outdoors; the alignment of delivery with formal (e.g., dedicated 9-week cycle) or informal (e.g., teacher decision to teach health on Mondays) health education cycles; and balancing Kids SIPsmartER with other health curriculum content. Teacher and principals also discussed factors related to the location where Kids SIPsmartER would be taught (e.g., gym or cafeteria versus classroom) and the number of students (e.g., teaching a whole PE class versus splitting girls and boys, bringing all 7th graders together during one period). Lastly, they discussed adaptations they considered or wanted to make, such as adding personal experience to make content more relevant and more hands-on activities.

Kids SIPsmartER’s fit

On the surveys, teachers and principals rated the importance of specific content areas and activities included in Kids SIPsmartER as important to very important. On the 5-point importance scale, teachers rated the content and activities as 4.61 (SD = 0.41) and principals as 4.38 (SD = 0.33). (Data not in table.) Based on ratings from completed fidelity checklists, teachers felt very confident delivering the lesson content and conducting the activities as 6.82 (SD = 0.36) on a 7-point agreement scale. Ratings were similar during both the first and second Teacher-Led Years. On average, teachers strongly agreed [6.33 (SD = 0.74) out of a potential score of 7] that students were engaged in the program. Rates were lower during the second Teacher-Led delivery cycles compared to the first, with ratings of 5.64 (SD = 1.11) and 6.54 (SD = 0.48), respectively.

Teachers and principals discussed ten factors that influenced their perceptions of Kids SIPsmartER’s fit. They identified six factors that were positively associated with its fit, two that were negatively associated, and two that were mixed. These factors are present in Table 4.

Of the factors positively associated with the fit of the Kids SIPsmartER classroom component two related to its compatibility with teaching and learning activities: recognition that Kids SIPsmartER filled a curricular gap and perceptions that its content and activities were age appropriate. Another reflects student engagement, as they reported students enjoyed its hands-on activities. Another factor that facilitated its fit relates to Kids SIPsmartER’s design as teachers reported felt supported by the implementation activities. Teachers and principals also mentioned seeing changes in student awareness and actions related to their beverage behaviors as a result of their participation. Lastly, teachers and principals perceived the focus of Kids SIPsmartER as addressing a regional health need.

The two factors negatively associated with Kids SIPsmartER’s fit reflected student engagement. Teachers reported that students disliked activities that were either very didactic or that the students did not perceive as relevant. Teachers in schools without a dedicated health education class (i.e., PE and health were the same class) reported that students disliked that Kids SIPsmartER, like other health education activities, took away from gym time.

The two factors that had both positive and negative impacts were related to compatibility with teaching and learning activities. Teachers and principals acknowledged that while Kids SIPsmartER reflected multiple standards of learning, it did not meet all necessary health education standards. Also, the impact of class size and location on perception of fit varied, depending on a school’s particular experience.

Discussion

Findings from this process evaluation of indicators related to the implementation of Kids SIPsmartER highlight that Kids SIPsmartER can be implemented by teachers following the initial Researcher-Led Year with moderate to high intervention fidelity. Furthermore, teachers engaged with the SISTER-aligned Kids SIPsmartER implementation bundle activities and perceived them positively. Lastly, findings demonstrate that while the Kids SIPsmartER intervention generally “fits” the needs of schools, teachers, and students, teachers and principals looked to increase the fit of Kids SIPsmartER by adapting delivery approaches to better reflect school-level factors and enhance the potential of student engagement. Together, the study findings have implications for the refinement of Kids SIPsmartER’s implementation strategy activities to support scalability and sustainment and for the larger literature related to implementation strategies in the context of school-based interventions.

Teachers reported delivering 97% of expected lessons as well as a high overall fidelity rate of 94% among returned checklists. However, even with direct reminders sent from research team members to teachers, approximately 25% of fidelity forms were not returned, thus overall fidelity could be as low as 66% if counted as zeros. Regardless, Kids SIPsmartER fidelity scores were within the range of lesson fidelity reported by EBBIs targeting energy-balanced related behaviors when delivered by teachers [33–35]. Nonetheless, this low return rate of fidelity forms by non-researcher delivery agents – which is not uncommon [36, 37] – muddles the interpretation of the fidelity. Therefore, other strategies, such as automated reminders and facilitation may need to be incorporated into the implementation bundle to promote return and/or identify barriers to their return in real-time.

Furthermore, Kids SIPsmartER’s implementation fidelity may have been affected as teachers adapted delivery approaches to better fit the context of their school. Among the executed adaptations to Kids SIPsmartER, some appear fidelity-consistent (e.g., incorporating personal anecdotes, incorporating video, adding supplemental activities) while others are potentially fidelity-inconsistent (e.g., condensing the timeline of Kids SIPsmartER by consistently delivering multiple lessons per week, reducing planned number of lessons to five). Adaptation is inevitable when maintaining or sustaining interventions outside of the initial research period as it can enhance contextual fit and not all drifts in intervention delivery are negative [38]. Yet, it is important to facilitate fidelity consistent adaptations to promote intervention fidelity and foster desired changes in behavioral outcomes [38]. This finding highlights the importance of ensuring the inclusion of implementation activities that would support teachers in making fidelity-consistent adaptations and/or executing strategies to overcome school-specific barriers.

As described earlier, Kids SIPsmartER incorporated multiple intervention activities into its implementation bundle that operationalized specific implementation strategies identified in SISTER [12, 13]. Some of these activities were directly related to facilitating Kids SIPsmartER’s intervention fidelity, including the manualized curriculum, lesson overview videos, formal training activities, and informal technical assistance meetings. Findings about the perceptions of these fidelity-related activities highlight that teachers found the activities to be useful and supportive and, with the exception of the information technical assistance meetings, often engaged in them, particularly during the first Teacher-Led Year. Findings also suggest teachers engaged with activities differently. Importantly, a crucial fidelity-related activity of the Kids SIPsmartER implementation bundle is the Researcher-Led Year during which researchers taught lessons with support of teachers. While watching and supporting the researcher deliver the intervention content during this initial year afforded teachers the opportunity to view faithful delivery, this activity could have had a meaningful impact on intervention fidelity during the subsequent Teacher-Led years. Yet, it would not be sustainable to continue to incorporate if Kids SIPsmartER were to be widely disseminated and implemented, due to high staffing and travel costs. Additionally, given teacher turnover, 45% of teachers who delivered Kids SIPsmartER during their school’s first or second Teacher-Led Year were either not present for or unable to engage in their school’s Researcher-Led Year. This suggests the need to adapt Kids SIPsmartER’s implementation activities to ensure all are scalable and resilient to teacher turnover, such as the incorporation of digital and/or asynchronous strategies.

Considering the broader evidence of teacher preferences for professional development allows for the identification of reasons that may have supported the teachers’ relatively positive reactions and moderate use of the Kids SIPsmartER implementation bundle activities. These activities were multidimensional, modeled, adaptable, and valued. These features have been identified as desired by teachers for professional development activities [39]. Also, Kids SIPsmartER’s implementation activities were designed to minimize additional time requirements. Time cost is a consistent barrier to teacher engagement [40, 41]. This barrier may be more relevant to PE/health teachers, as compared to teachers of other subjects, they are less likely to get release time for professional development activities [42]. While determining the direct impact of the use of these strategies on intervention fidelity is outside the scope of this study, specifically understanding what drives intervention fidelity for Kids SIPsmartER is an important future research step. However, understanding this impact would be necessary to refine its implementation bundle in order to bring the intervention to scale and is a next research step for the Kids SIPsmartER intervention.

Additionally, findings demonstrate that Kids SIPsmartER’s classroom component – both its curricular materials and the implementation bundle – meets the needs of schools and teachers and reduces many common barriers that inhibit the implementation of EBBIs in schools [7, 9, 10]. Importantly, these positive perceptions about Kids SIPsmartER’s classroom component reflect constructs from all five domains from the Consolidated Framework for Implementation Research (CFIR) [43]. This underscores that Kids SIPsmartER and its implementation strategy bundle facilitate its delivery across multiple levels within the school context. Specifically, Kids SIPsmartER was viewed as important as it was relevant to both the needs of the broader PE/health curriculum which is set at the state level through the standards of learning and reflected a health behavior that viewed as impacting students’ health in a meaningfully negative way (Outer Setting domain). Also, teachers felt the program filled an important curricular gap, viewed the curricular and training materials as being high quality, reported the curriculum was easy to deliver, and felt supported and confident they could deliver the content (perceptions cut across Innovation, Inner Setting, Characteristics, and/or Implementation Process domains). It is necessary to place these findings in the context that health education is often relatively unstructured due to the lack of testing for related educational standards and teachers often have to develop or piece together their own lessons. Teachers also viewed the program as being generally well received by students (Innovation, Characteristics domains). Taken together, these findings suggest that Kids SIPsmartER provided teachers with multiple motivators and supports to deliver the program and deliver it faithfully. Notably, teachers’ perceived value of a nutrition education focused EBBI is used to balance barriers and support implementation [44]. Past evidence demonstrates that what is valued about an EBBI varies by teacher and ranges from the potential public health benefit, to whether it is well-received by students, to how the program aligns with a school’s larger pedagogical aspirations [44, 45]. Furthermore, evidence supports that facilitator (i.e., teacher) interest in an intervention and fidelity are positively associated [46, 47]. Therefore, ensuring EBBIs and their implementation bundles foster teacher motivation by addressing facilitators at multiple levels to deliver the program can help bolster fidelity.

Overall, the findings from this study suggest the Kids SIPsmartER implementation bundle has multiple strengths that supported its maintenance, including its varied activities (e.g., comprehensive manualized curriculum, teacher-facing videos, in-person onboarding) and its flexibility that allows for adaptation based on school-level factors. These strengths led Kids SIPsmartER and its implementation activities to be viewed as useful and consistently engaged with. The implementation bundle could be strengthened to support fidelity and maintenance by intentionally ensuring incorporated implementation strategies (i) support fidelity consistent adaptations, (ii) are adaptive to specific needs of schools and teachers, (iii) are resilient to teacher turnover and changing teacher needs over time (i.e., differences between first and second years of delivering), and (iv) are scalable. Adding a structured facilitation component to the implementation bundle may be a pragmatic way to address these considerations. Facilitation is a set of strategies used by a facilitator to support delivery agents in implementing an EBBI. The i-PARIHS framework posits it as the “active ingredient” that brings together an EBBI (innovation), the recipients, and context in order to achieve successful implementation [48]. The current Kids SIPsmartER implementation bundle incorporates established facilitation processes, including (i) benchmarking via fidelity checklists and communication strategies, (ii) mentoring through the informal technical assistance meetings, and (iii) leveraging influence by keeping principals and district administrators informed. Yet, activities related to these processes could be strengthened. For example, feedback to teachers could be formalized, clearly focused on successes and barriers, and incorporate data from fidelity checklists. Likewise, principal influences to support implementation fidelity could be leveraged more consistently and in manner that is non-punitive. Additionally, intentional incorporation of action planning by delivery agents, another facilitation process, would allow for the other considerations to be directly addressed. Importantly, future implementation-focused studies of Kids SIPsmartER will incorporate the lessons learned from this study to inform the development of an Implementation Research Logic Model [49]. This model will allow for the linkages among implementation strategies, the individual and school-level determinants that impact their execution, and the implementation and effectiveness outcomes they could impact to be clearly articulated, informing the evaluation design of subsequent trials.

Study findings should be interpreted with consideration of three primary limitations. First, generalizability could be limited due to the included schools being in one rural region. However, schools came from two states and four school districts. Furthermore, included teachers had various degrees of teaching experience (i.e., first-year teachers and teachers with 20 years of experience). Second, fidelity ratings are based on teacher self-report, as there were limited observations due to COVID-19 restrictions during SY 2020–2021 and start of SY 2021–2022. Third, findings are limited to descriptive data, as this is a secondary analysis of implementation data from a type 1 hybrid effectiveness-implementation trial and there was inadequate power to conduct inferential analyses to determine the impact of engagement with implementation strategies on fidelity. These limitations should be considered alongside the study’s strengths: the mixed-methods design and having the interviews conducted by research staff who had limited interaction with the teachers they interviewed outside of the interviews. This latter approach might have reduced the social desirability of responses.

Conclusions

Findings from this mixed methods exploration of the teacher-led delivery of Kids SIPsmartER, a promising intervention, highlight how implementation protocols integrating multiple established implementation strategies can support the maintenance and sustainment of EBBIs in schools. Such protocols can not only ensure fidelity in delivery but also foster teacher motivation when tailored to meet the unique needs of both educators and schools. Furthermore, study findings underscore the importance of ensuring broader dissemination by aligning implementation bundle activities with fidelity-consistent adaptations to enhance contextual fit, addressing school-level barriers, and ensuring scalability. Collectively, findings will be used to adapt the Kids SIPsmartER implementation strategies and to empirical text the adapted strategies in a fully-powered cluster randomized controlled trial focused on implementation fidelity in rural Appalachian middle schools.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Abbreviations

CFIR

Consolidated Framework for Implementation Research

EBBI

Evidence-based behavioral intervention

PE

Physical education

SISTER

School Implementation Strategies, Translating ERIC Resources

SSB

Sugar-sweetened beverage

SY

School-year

Author contributions

Study conception and design (KJP, JMZ); Data collection (KJP, ALR, TM, BMK); Data analysis (KJP, TM, BMK, HGW, CH); data interpretation (KJP, JMZ, ALR); drafted the manuscript (KJP); provided feedback (JMZ, ARL, BMK, TM, HGW, CH, LS). All authors have approved the final version.

Funding

This study was funded by the National Institute of Health (NIH), National Institute on Minority Health and Health Disparities (R01MD012603, PI: Zoellner).

Data availability

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Declarations

Ethics approval and consent to participate

Study procedures were approved by the University of Virginia Institutional Review Board for the Social and Behavioral Sciences. This research was conducted in accordance with the principles of the Belmont Report and the Declaration of Helsinki. Informed consent was waived by the University of Virginia Institutional Review Board for the Social and Behavioral Sciences.

Consent for publication

N/a.

Competing interests

The authors declare no competing interests.