Pragmatic adaptations to Kids SIPsmartER’s implementation protocol supported its delivery during the COVID-19 pandemic

Abstract

Objective:

Describe the adaptation, implementation, and perceptions of Kids SIPsmartER’s classroom component during the COVID-impacted 2020–2021 school year.

Design:

Mixed methods process evaluation.

Setting:

Seven rural Appalachian middle schools (United States).

Participants:

Middle school teachers (n=14) and principals (n=6).

Intervention:

Kids SIPsmartER was a multi-level, school-based intervention designed to decrease sugar-sweetened beverage intake. The 12-lesson classroom component was supported by an implementation protocol.

Measures:

Implementation protocol adaptations, program perceptions, and the school context were assessed using teacher and principal interviews, teacher-completed fidelity checklists, and researcher-maintained field notes. Adaptions were mapped to the Framework for Reporting Adaptations and Modifications-Enhanced (FRAME).

Analysis:

Qualitative data were content coded. Quantitative data were summarized using descriptive statistics.

Results:

All schools maintained Kids SIPsmartER and delivered 100% of lessons. Ten adaptations were made to the implementation protocol. Schools used adapted delivery approaches to meet individual needs. Teachers and principals identified more benefits than barriers to implementing the program.

Conclusions and Implications:

Utilizing a strategically adapted implementation protocol that was flexible to schools’ individual needs allowed all middle schools to deliver Kids SIPsmartER during the 2020–2021 school year. Findings identify adaptation considerations that other school-based evidence-based interventions could incorporate to facilitate delivery during high-stress times.

INTRODUCTION

The implementation of evidence-based interventions or programs (EBIs) that target behavior change, such as nutrition education interventions, in schools is challenging. The implementation of school-based EBIs add burdens to schools both in general and specifically related to time (e.g., time for training, preparation, and program administrative tasks) and obtaining available resources. ¹ Furthermore, as EBIs are not a standard part of school curricula, they compete for instructional time with other programs, some of which may be perceived as being more aligned with standards of learning. ^2,3

Unsurprisingly, the COVID-19 pandemic further complicated the implementation of EBIs. Changes to K-12 instructional, safety, and building protocols in the Spring of 2020 and throughout the 2020–2021 school year added additional burdens for school staff, creating a unique high-stress year. These changes included decreased instruction time, a shift to virtual instruction, modifications of in-person instructional approaches to limit contact between teachers and students, and restricting the ability of non-school personnel to enter schools. ^4–7

Yet, the implementation of EBIs in schools can be fostered in ordinary times of challenge and in extraordinary ones, such as a global pandemic. This can be achieved by ensuring the fit of the EBI with the school’s wants, capacity, and needs and establishing communication channels to support delivery. ^2,8 Also, providing reasonable flexibility is particularly crucial for supporting the short and long-term implementation of EBIs, both in schools and other settings. ⁹ Being able to foster the implementation of nutrition education focused EBIs during times of challenge, including unique high stress times, is important. Nutrition education focused EBIs target behaviors that impact multiple preventable chronic diseases that have significant population health impacts, including obesity and diabetes, and that are present regardless of other challenges schools are facing. ¹⁰ Also, these other stresses may exacerbate unhealthy nutrition related behaviors and impact rates of the chronic diseases, such as was seen during the COVID-19 pandemic. ^11–13

Adaptations are a means to provide this flexibility. They are deliberate and strategic modifications to an intervention component or protocol. ^9,14 Adaptations take a number of forms: tailoring materials; changing delivery modality (e.g., from in-person to virtual); adding, removing, or reordering components; and loosening structure (e.g., reducing requirements for number of activities). ¹⁴ Importantly, adaptations can increase reach, engagement, retention, and fit and reduce program drift. ⁹ The Framework for Reporting Adaptations and Modifications-Enhanced (FRAME) provides a systematic approach for describing adaptations to EBIs. As this framework allows for the types, reasons, and decision making around these adaptations to be clearly documented, these changes can be more readily presented and shared. ^14,15

Yet, existing literature about the adaptation of EBIs for schools primarily provides theoretical guidance for making and/or reporting adaptations. Some of this guidance comes from evidence-driven models and frameworks (e.g., the Adaptome, ⁹ FRAME^14–16). Other guidance comes from reviews^2,17 and mixed methods or qualitative studies⁸ exploring what schools want and need to adopt, implement, and maintain EBIs. However, there is limited evidence about how these adaptations have been made and responded to in “real-time” as a response to changing school needs. Likewise, there are no known studies about the adaptation of nutrition education focused EBIs in the school classroom context.

Kids SIPsmartER was a 6-month multi-component intervention designed to decrease sugar-sweetened beverages among middle school students and their caregivers. ¹⁸ It was guided by the Theory of Planned Behavior (TPB), ¹⁹ health and media literacy concepts, ^20,21 and the RE-AIM framework. ²² Kids SIPsmartER consisted of a 12-lesson classroom-based curriculum for 7^th grade students as part of their physical education and health class and a short message service (SMS) component for caregivers. The curriculum was delivered over 7 months, with lessons designed to be taught in-person over a single approximately 40-minute class session and with the support of PowerPoint slides and a student workbook. Starting during the 2018–2019 School Year, the effectiveness of Kids SIPsmartER was assessed through a cluster randomized, type 1 hybrid effectiveness-implementation trial. Both the intervention and the trial design are described in more detail in the Methods and elsewhere. ¹⁸ Study findings demonstrated statistically and clinically significant reductions in SSB intake among intervention students, relative to control students, as well as greater reductions among students who were high SSB consumers at baseline. ²³

During the COVID-impacted 2020–2021 school year, 7 rural Appalachian middle schools were enrolled in Kids SIPsmartER. This article describes the organizational outcomes (i.e., implementation protocol adaptations, curriculum adoption, and curriculum implementation) related to Kids SIPsmartER’s classroom curriculum component in these middle schools during this school year. To contextualize the school environment during this year, the instructional experience of physical education (PE)/health teachers is first described. The purpose of this process evaluation is to extend understandings of how adaptations to school-based EBIs during COVID supported their adoption and implementation. Findings may be generalizable to other high-stress periods often faced by teachers and schools.

METHODS

This study was a concurrent mixed methods process evaluation²⁴ of the delivery of the on-going Kids SIPsmartER trial during the 2020–2021 school year. In an expedited review process, study procedures were approved by the University of Virginia Institutional Review Board, including the semi-structured interview scripts for teachers and principals. For teachers and principals, consent was implied by agreeing to participate in the study’s implementation activities and by completing the interviews. Also, caregivers provided written or oral consent and students provided assent, yet data is not reported in this article.

Kids SIPsmartER Intervention Design and Implementation Protocol

The 5-year, type 1 hybrid effectiveness-implementation and cluster randomized control trial to evaluate Kids SIPsmartER was launched in 2018 (Clincialtrials.gov: NCT03740113; 2018–2022). In brief, a type 1 hybrid is primarily designed to address aims pertaining to an intervention’s individual-level effectiveness and to also explore organizational-level implementation outcomes. ²⁵ Schools were randomized into either the intervention condition or delayed contact control condition during their first year of involvement in the research trial. Intervention effectiveness (i.e., change in student sugar-sweetened beverage intake) was the primary outcome during this first year (data reported elsewhere). ²³ After this first year, schools could continue to implement Kids SIPsmartER, with outcome and process data collected to explore the trial’s secondary organizational level aims (i.e., adoption, implementation, and maintenance), which is the focus of this article. Importantly, during the 2020–2021 school year, all enrolled schools had completed their first year of involvement, which offered a unique opportunity to allow adaptation without impacting the primary effectiveness aim of the trial.

Prior to the 2018–2019 school year, the research team developed an implementation protocol to support adoption and implementation of the programmatic and research components of Kids SIPsmartER. This protocol was developed using established implementation strategies from the School Implementation Strategies, Translating ERIC Resources (SISTER). ^26,27 The specific portions of the implementation protocol to support the classroom component are the focus of this article. Table 1 details the pre-COVID implementation protocol for the classroom component specific to (1) training and technical assistance (e.g., in-person trainings, co-teaching during first year of implementation, regular communication), (2) lesson delivery (e.g., in-person lesson delivery, physical student workbooks, set implementation calendar), and (3) process evaluation (paper fidelity sheets, lesson observations). It also describes the adaptations to this protocol, which are presented in the Results section. Notably, while the research team encouraged teachers to adhere as closely as possible to the implementation protocol when delivering Kids SIPsmartER, there were no specific adaptations that were deemed acceptable or not. This was due to the uniqueness of the circumstances and the unique needs of each school.

Table 1:

Adaptations to Kids SIPsmartER Classroom Component Implementation Protocol during the COVID-Impacted 2020–2021 School Year (SY)

Expected Implementation Related Activities Pre-COVID (Before 20–21 SY)	Adaptations for COVID-Impacted 20–21 SY	Adaptation Description Using Elements of the FRAME^15
		Reasons	Goals	Nature of Adaptation	Level Impacted
Training and Technical Assistance
Half-day in-person pre-SY staff development	Multiple mini virtual staff developments	COVID protocols, reduce burden	Sustain engagement	Change mode	Teacher
Co-teach with researcher during 1st year of program delivery	Removed co-teaching option	COVID protocols	Sustain delivery	Remove element	Teacher
Communicate weekly with research team (teachers)	--	--	--	--	--
Communicate weekly with research team (principals)	Reduced frequency of contact	Reduce burden	Sustain satisfaction	Loosen structure	Principal
Lesson Delivery
Co-teach with researcher during 1st year of program delivery	Removed co-teaching option	COVID protocols	Sustain delivery	Remove element	Teacher
Solo deliver program after 1st year of program delivery	Solo teacher deliver in 1st year of program delivery	COVID protocols	Sustain delivery	Loosen structure	Teacher
Deliver 12 lessons in-person	Added virtual delivery option	COVID protocols	Sustain delivery	Change materials	Student, Teacher
Lessons supported by physical workbooks and PowerPoint slides	Added student facing videos and electronic workbooks	COVID protocols, reduce burden	Sustain delivery	Change materials	Student, Teacher
Set implementation calendar for Fall and Winter	Flexible delivery timeline	COVID protocols, reduce burden	Sustain delivery	Loosen structure	Teacher
Process Evaluation
Complete paper fidelity sheets for each lesson	Electronic fidelity forms that also can virtual delivery	COVID protocols, reduce burden	Sustain engagement	Change mode	Teacher
3 in-person lesson observations per teacher after Year 1	Phone-based observations of “in-person” lessons	COVID protocols	Sustain engagement	Change mode; loosen structure	Teacher

^{^}Framework for Reporting Adaptations and Modifications-Enhanced

Participants

During the 2020–2021 school year, Kids SIPsmartER was delivered in 7^th grade physical education and health classrooms in 7 middle schools in 3 rural counties in Appalachian Virginia. ²⁸ These counties score “low or very poor” on the Health Opportunity Index. ²⁹ During this school year, 824 students received Kids SIPsmartER as part of their physical education and health class.

Six of the 7 principals had been involved with Kids SIPsmartER the previous school year; 1 was new to the school. Of the 14 physical education and health teachers who taught Kids SIPsmartER this school year, 9 (64%) had previously delivered the program and 5 (36%) had never delivered the program (i.e., were in a delayed contact school the previous year or were new to the school).

Instrumentation and Procedure

This process evaluation incorporated 3 data sources. These sources were semi-structured teacher and principal interviews, field notes, and fidelity checklists.

Using a semi-structured interview protocol developed for this trial by the study team, the lead author conducted interviews twice among teachers (mid-year and end-year) and once among principals (end-year). Interviews were audio-recorded with the approval of each interviewee. The interview protocols consisted of 2 sections: (1) how COVID-19 impacted instructional delivery in their school in general and in terms of their role as physical education and health teachers and (2) their specific experiences delivering Kids SIPsmartER, including decisions about delivery, barriers to delivery, perceived benefits, and program fit. The mid-point and end-of-the-year protocols included 14 and 10 questions, respectively. Each question included specific probes.

The research team-maintained field notes were documented throughout the year. Research team members were Masters or doctoral-level trained health professionals (e.g., registered dietitian nutritionist (RDN), Master of Public Health (MPH)). As part of the field notes, they logged interactions with teachers related to training and technical assistance activities, timing and content of regular correspondence, when teachers delivered lessons, reasons for delays, and specific concerns. Also, these field notes captured adaptations made prior to the start of the 2020–2021 school year and rationales for the adaptations.

Separate from field notes, Teachers and research team members were to complete lesson specific fidelity checklists after each delivered or observed lesson. These checklists collected data about when a lesson was delivered, mode of delivery (in-person, virtual, hybrid), perceptions of student engagement [7-point scale ranging from (1) “strongly disagree” to (7) “strongly agree”], and other notes about delivery. It is important to note that for in-person and hybrid delivery, fidelity checklists assessed the degree to which specific lesson activities were completed. Teachers indicated if they completed, partially completed, or did not complete each activity. They also identified if they modified an activity. However, these data were not incorporated into this article due to the high frequency of virtual lessons for which the degree of degree of lesson activity completion was not assessed due to the nature of their delivery. For virtual lessons, teachers did not deliver lesson content but rather uploaded the student-facing videos (which had 100% fidelity to lesson activities) and workbook pages with the expectation that students would complete the virtual lessons asynchronously.

Data Analysis

Interview recordings were transcribed verbatim, cleaned, and de-identified by the research team. NVivo (version 12.0, Lumivero LLC, Denver, CO, 2017) was used to manage the qualitative data. Transcripts were coded using a hybrid of directed and conventional content analysis. ³⁰ First, transcripts were coded into domains reflecting (1) implications of teaching during the 2020–2021 school year, (2) Kids SIPsmartER delivery, (3) perceived benefits to Kids SIPsmartER, and (4) perceived barriers to Kids SIPsmartER. Next, 2 members of the research team reviewed passages within each domain to identify emergent codes. Codes were discussed with a third research team member and a list of emergent codes was created. The codes were defined and added to the codebook. Then, passages within each domain were coded into relevant codes. Two researchers coded transcripts independently and met to resolve differences, with the third researcher settling any discrepancies. Codes related to benefits were then organized into higher-level categories following a similar process.

Field notes were reviewed to identify adaptations of curriculum implementation activities. Identified passages were content coded to 4 of the 8 elements of the FRAME: (1) reasons for adaptation, (2) nature of adaptation, (3) goals of adaptation, and (4) level impacted. ¹⁵

The number of lessons delivered per school and type of lesson delivery were tallied using field notes and data from the fidelity checklists. Average scores across lessons and schools were calculated for teacher ratings of perceived student engagement in lessons.

RESULTS

Teacher and Principal Completion of Interviews

Thirteen teachers (93%) and 6 principals (86%) completed interviews. Interviews were completed with at least 2 representatives from each school. Ten (70%) teachers completed both mid-year and end-year interviews.

Context of Teaching during the 2020–2021 School Year

Teachers and principals reported 8 factors across two categories that impacted the context of the delivery of physical education and health classes during the 2020–2021 school year. The 8 identified factors are described in Table 2, along with exemplar quotes and the frequency by which they were mentioned by teachers and principals.

Table 2:

Context of Teaching during the COVID-Impacted 2020–2021 School Year

General School Context
The year was hard and completely different from any previous school year.	6 (46%)	4 (67%)	“This is my 30th year here and this is the hardest year I’ve ever had. It’s just like my first year teaching. It is like starting over because we had to do everything different.” (Teacher B) “… I mean, you know, not being able to gather together. I mean, you have to you have to rethink lunches. You have to rethink class numbers. You have to rethink any kind of extracurricular or, you know, any kind of academic gathering that you would normally have. You know, all those things you have to have a different plan for. Plus, you got to follow whatever the divisions or mitigation plans or, you know, when you’re having those events. I mean, it’s just been tough.” (Principal 3)
There was (perceived) instability due to regular shifts between in-person, virtual, and hybrid instruction.	11 (85%)	3 (50%)	“We’re coming back in person on Monday, the 25th so we’ll be coming back in person for the second time this year. We were in-person for 40 school days back during the fall. And that was four days a week with virtual learning on Wednesday. Now we’re virtual learning. And starting next week, we’re going back to five day a week, but we still will have a healthy number of students that will remain virtual and they’ll remain virtual for the remainder of the school year.” (Teacher K) “It was probably October before we. [Yeah], I think in October we got some kids back in the building on a rotating basis. [OK]. We started back doing like we have an A day and a B day … And then, … [w]e did it for a while. Then we went back virtual for a little while. Then we got to come back and we got four days a week here.” (Principal 5)
Teachers had less time with each class due to schedule changes to accommodate safety protocols and hybrid instruction.	11 (85%)	2 (33%)	“Now that we’re ending at two o’clock, our 6th and 7th grade classes, which are in the afternoon, they’re only, I guess are only 30 minutes long, which has been kind of tough. ... By the time we get in here, every kid situated, we take attendance, which we have to do attendance for the kids that are in person and the kids that are on the meeting. And they don’t always show up right at the beginning of class. So once we do that, I think we have roughly 20 minutes to get a lesson done.” (Teacher M) “The biggest impediment was probably they came to us and told us, rather than having a full day of school, we would release at one o’clock, and that instead of having our students [for] 8 periods … we would see the students four days a week and we would be A/B days. So, we went from having 90 minutes with students in math and reading and 50 minutes twice instead of 90 minutes every day, 50 minutes, twice a week with them...just the lack of time has been the biggest problem.” (Principal 4)
Teachers (and students and schools) had to quickly adapt to technology-based instruction.	9 (69%)	3 (50%)	“But no, I guess the thing that has changed the most for me, you know, I check my email and stuff on my computer, but I didn’t do much from a computer where Now as I have to do pretty much everything from a computer.” (Teacher J)
Context Specific to Physical Education and Health Class
Teachers shifted their expectations for student learning and engagement in physical education and health.	12 (92%)	2 (33%)	“I think the expectations are still there that we want the kids to learn at a high level and we expect a lot from them. But, … it’s a little more lenient this year as far as turning in work for the online kids. In-person kids, we’re there with them and can keep them up. … So, there’s a give and take.” (Teacher E)
The delivery of physical education and health looked very different. • Instructional approaches changed to fit COVID protocols, virtual learning, and shortened classes. • Evenness in instruction had to be maintained across groups • Teachers identified priorities to “best” utilize in-person class time and meet instructional goals	11 (85%)	0 (0%)	“Well, with our gym classes, … they told us that we can’t do very much of anything where we touch an object … And we’re just very limited.” (Teacher D) “… [T]hey told us that everything we did for the online kid we had to do for the in-person key and vice versa. So, if the online kids were getting that instruction ... on the video that we want to make for sure, the in person got it also.” (Teacher A) “Well, I remember we talked about at the beginning, we weren’t going to start it right off. That was a big thing, was getting all the physical fitness of being outside as long as we could. So that was a big part of it. I guess. we’re trying to do.” (Teacher G)
Teacher-student relationships changed. • Teachers felt less connected with their students. • Students were seen as less engaged, especially when virtual. • Students’ needs changed due to negatively impacts of COVID on education and other facets of daily life.	11 (85%)	0 (0%)	“[I]t’s been weird, too, for me, because I don’t have all my kids and it’s like I was talking to one teacher, if anything were to happen, … I would have never met half of my students. I met one Monday night, at softball practice, who is an online student. And I was like, oh, by the way, you know, I’m your gym teacher. So nice to meet you. Six months into school. So that’s very, very different for me, too, not knowing all of my students.” (Teacher G) “I mean, it’s been a difficult year of getting kids to do anything. So the program is set up in a good way that was easy to give to the kids. The hard part was getting them to do it and I guess to have accountability for that.” (Teacher F) “I’m just very sensitive to the fact that the kids are scared and that there are some days [there] may be more important things than finishing every single thing that I have planned for them and doing some teachable moments and make sure that making sure that if they have questions that if I don’t know the answers, that I find a reputable place that I can find them answers.” (Teacher D)
Physical education and health teachers took on new tasks and roles.	6 (46%)	1 (17%)	“But we’re taking temperatures, passing out hand sanitizer, doing all that stuff in the morning.” (Teacher G) “Typically, the school day started at eight o’clock and now we’re opening our doors at seven thirty and we can’t put the kids in the gym, so they all go directly to 1st or 5th period, depending on the day. [We]’re expecting the teachers to be here for 20 minutes earlier.” (Principal 4)

The first 4 identified factors reflected those that impacted the entire school. These factors were identified by at least a third of teachers and principals. First, teachers and principals noted that the entire year was difficult. The teachers and principals, including those with decades of experience, discussed the year as being like it was their “first year teaching.” They also highlighted logistic changes that had to be made to common activities, such as the distribution of lunch. Second, interviewees reported feeling a lack of stability. In large part this was due to a constant flux in mode of instruction between in person, virtual, and hybrid instruction as new COVID cases occurred and school needed to be shutdown. Third, there was less overall instructional time due to schedule changes to accommodate safety protocols and/or hybrid instruction that reduced time spent in overall instruction. Class periods were shortened sometimes by half and/or students only had in-person instruction 2 days a week. Fourth, teachers, along with students and the individual school districts, had to quickly adapt to new instructional modalities. This learning curve was particularly steep as prior to March of 2020 none of the schools involved in Kids SIPsmartER had provided students with laptops or used on-line learning platforms. Therefore, they had to build infrastructure for virtual learning (e.g., getting students laptops, ensuring student Wi-Fi access at home), shift their teaching approaches, and workflow (e.g., checking emails).

The other 4 factors were specific to the context of the physical education and health class. Teachers discussed all these factors, while principals only mentioned 2. This is unsurprising given the context of these factors is specific to the execution of physical education and health. One, teachers shifted expectations for student learning to balance accountability with the reality of the instructional/learning climate. For example, strictness on deadlines was relaxed. Also, because of the more flexible nature of physical education class, some physical education teachers used part of their class to let the children talk to one another, instead of having delivering content. Two, COVID safety protocols significantly shifted what physical education and health class looked like, as they required teachers to alter their usual in-person instructional approaches, create virtual lessons, and deliver more health education lessons. This factor was only identified by teachers. Importantly, teachers discussed needing to maintain evenness in instruction between groups (i.e., in-person and virtual students). This requirement was often for the same day versus over the course of the week when delivery was hybrid. These changes required teachers to set clear priorities to make best use of their class time. Three, teachers identified that teacher-student relationships changed. Teachers, who talked about how they could be more connected to many students, discussed feeling less connected with their students. They also noted that their students were less engaged, with many students were not completing homework and other activities. Four, physical education and health teachers discussed the new tasks they were required to take on that impacted their teaching experience. Some occurred within their classrooms (e.g., taking student temperatures, cleaning surfaces), which reduce instructional time. Others were outside the classroom and unrelated to their physical education and health teacher roles (e.g., packing lunches for virtual students), which shifted their overall function in the school.

Adaptations to Kids SIPsmartER Classroom Component Implementation Protocol for the 2020–2021 School Year

Ten adaptations were made to the implementation protocol. These adaptations were made with input from teachers and school leadership during the preceding summer and at the start of the school year. Some were made prior to program implementation based on existing or expected changes to the school context, whereas others were made in reaction to changing protocols and/or policies of the schools and/or the University of Virginia. Some adaptations, including the electronic workbook pages and flexibility of the delivery timeline, were further refined during the school year based on the needs of the schools. These adaptations are described in Table 1. While adaptations were also made to the outcome evaluation aspects of this trial (i.e., shifting mode of delivery of consent forms and student and caregiver surveys), discussing these adaptations is beyond the scope of this paper.

Reasons for the adaptations to Kids SIPsmartER’s implementation protocol were 2-fold: (1) to accommodate COVID-related protocol changes that instituted virtual and/or hybrid learning and that restricted visitors and in-person meetings and (2) to reduce burdens on teachers who were having to adapt to new teaching methods (e.g., virtual learning) and create new lesson plans that were aligned with virtual learning and/or met safety protocols. Examples of adaptations included removing the co-teaching approach for teachers in their first year of delivery; allowing virtual delivery of lessons; creating resources to support virtual learning (e.g., student facing videos, electronic workbooks); adding flexibility into the delivery timeline to reflect the schools’ adjusted schedules; and changing the mode of training and process evaluation to be compatible with virtual communication approaches. The goals for the adaptations were to sustain teacher engagement, lesson delivery, and teacher satisfaction while maintaining the key learning objectives of the Kids SIPsmartER lessons. The resulting adaptations involved changing mode of delivery, removing elements of the implementation protocol, loosening the structure of the implementation protocol, and changing packaging or materials. The changes were primarily directed at activities conducted by teachers.

Adoption and Delivery of Kids SIPsmartER during the 2020–2021 School Year

Adoption.

All 7 schools involved with Kids SIPsmartER during the 2019–2020 school year maintained their involvement with the program during the 2020–2021 school year. Each of these schools delivered all 12 Kids SIPsmartER lessons to their 7^th grade students.

Completion of fidelity forms.

Teachers completed 73% (SD=38%, range: 0% to 100%) of the expected fidelity forms. Five virtual observations were conducted at 4 of the 6 schools that delivered lessons in-person.

Lesson delivery.

For the lessons in which the mode of instruction was reported (n=79) through fidelity checklists, 43% (n=34) were conducted as virtual lessons only and 57% (n=45) were conducted by teachers using a hybrid approach (i.e., lesson delivered both in-person and virtually). Of the hybrid lessons, 53% (n=24) were conducted synchronously (i.e., virtual students received the lesson at the same time as in-person students). The remaining 47% of hybrid delivered lessons were delivered asynchronously, with teachers uploading the student videos and workbook pages to the virtual classroom for students to complete and then delivering the lesson directly to their in-person students. Six schools utilized a mixture of in-person and virtual delivery approaches while 1 school delivered all 12 lessons using virtual instruction. Findings from the fidelity checklists were mirrored by the implementation notes and teacher interviews, both of which indicate that Kids SIPsmartER was delivered in multiple ways at most schools and that students received Kids SIPsmartER as a mix of in-person and virtual lessons.

In their interviews, teachers reported enhancing student learning and engagement during virtual instruction by tailoring the adaptations to their context. Teachers added aspects to their Kids SIPsmartER lesson delivery. All teachers reported assigning work to their virtual students for at least some Kids SIPsmartER lessons that was to be returned for, at minimum, a participation grade. Most reported having short follow-up conversations with students during in-person days about lessons that were delivered virtually. A few created short how-to videos to explain the lessons. However, during interviews, teachers mentioned not being able to hold students accountable for this work due to lack of accountability structures at the school level and prioritization of core subjects that are part of state testing over physical education and health.

Perceptions of student engagement.

Teacher ratings of student engagement were 5.1 (SD=0.87) for students receiving the lessons virtually and 6.5 (SD=0.61) for students receiving the lessons in-person.

Reported Benefits and Barriers to Implementing Kids SIPsmartER during the 2020–2021 School Year

Overall, teachers and principals stated that Kids SIPsmartER was delivered as well as it could have been given the circumstances and that the adaptations aided in this. However, teachers also perceived the execution and potential impact of the program to be lower than in previous years due to the impacts of COVID-19 on instruction. They also identified specific benefits and barriers to implementing Kids SIPsmartER during this school year, which are presented in Table 3 and described below.

Table 3:

Benefits and Barriers to Implementing Kids SIPsmartER during the COVID-Impacted 2020–2021 School Year Identified by Teachers and Principals

Benefits	Barriers	Exemplar Quotes
Perceptions of Impact
42% (n=8)
• Perceived as improving knowledge, awareness, and/or behaviors related to sugar-sweetened beverages. (general)		“… I think for the most part, I think kids are taking something away from this and are a lot more cautious like because that.” (Teacher L) “One of the things that they did say is that they [teachers] have recognized that a few kids are making better decisions when it comes to drinks: the things that they do drink and just what they bring to class with them. … They [teachers]said they have seen them have more bottles of water.” (Principal 3)
Conceptual Fit
89% (n=17)	26% (n=5)
• Perceived as a general positive addition to the health curriculum. (general) • Meets instructional needs for the school by updating health education materials and filling gaps in health education curriculum. (general, COVID-specific) • Addresses a relevant health need for the region, rural communities, and the nation. (general)	• There are other concepts needing to be addressed in health class (general) • Perceived as less relevant for low or non-consumers of sugar-sweetened beverages. (general) • Some content not perceived as relevant for students (e.g., financial implications). (general)	“We appreciate so much this program and any other programs that we can get that are up to date. Having a 1999 health book to work from …. I just think that’s ridiculous.” (Teacher D) “We don’t really have a health programs. We’ve got a 6th grade program. We don’t really have anything for 7th grade. So this has been something that we can use, especially now where we’re not doing as much PE [physical education] stuff.” (Teacher J) “I would say very important because that falls back on, you know, it’s not only a problem here. You know, our country and our nation has an obesity problem. And that’s on the news every other time you turn it on.” (Principal 3) “… we do have other things that we have to do in health. And just depending on how we set that up for our school year or whatever, how many lessons that would need to be taught in a variety of different areas.” (Teacher F) “And, you know, what the problem is, in my opinion, and there are several young ladies in both of my classes …. They are not allowed to have any sugary drinks. … I’ll tell them just, you know, just fill in that you didn’t have anything [on the tracking sheet]. … But sometimes I think they think they’re just sitting there and it’s not as relevant to them.” (Teacher D) “That’s [financial implications] important. But, you know, with middle schoolers, that’s something they don’t even think about. ... And that’s just one of the things I think that comes with maturity and as they get older.” (Principal 3)
Materials and Lesson Design
89% (n=17)	11% (n=2)
• Content is age-appropriate, relevant, and delivered in engaging ways. (general) • Easy to deliver because materials are well-designed, compatible with instructional needs, and were adapted to support virtual delivery. (general, COVID-specific)	• The organization of the student workbook creates some confusion during the lessons. (general) • Lessons are too long. (general)	“ I think most of it’s been pretty relatable to the kids, I taught business and marketing for 8 years before I was full time PE teacher. So I really like the advertising portion, … And the activities in the lessons are, like I said, they’re very relatable to the kids.” (Teacher M) “We started out talking about the material. And, you know, a lot of times when you when you get some material on things, it’s not it’s not up to date and it’s not relevant to kids. But … [what is] good with the program is this the material is updated. The kids can relate to it fairly easily.” (Principal 3) “I think the lessons and the way that it’s presented, I really like [it] ... we’ve never taught the material before or watch someone do the material. But I’m really liking the videos where it’s clear and concise and explains the chapter. And I think the lessons are good and beneficial. That was my only thing of suggestion is the book. [OK] of how it’s ordered. (Teacher L) “They are getting it twice a week. The lessons that you all have are 35 – 45 minutes. So I just basically said, cut it in half. … they’re sitting in seats, you know, seven hours a day or, you know, five hours a day. And I want them moving every day. (Principal 7)
Program Implementation
	47% (n=9)
	• Delivery was disjointed due to fluctuation between in-person and virtual instruction. (COVID-specific) • The teaching space was not ideal. (general, COVID-specific) • Had limited instructional time for PE/Health due to schedule changes. (COVID-specific) • There were technology barriers related to virtual instruction. (COVID-specific)	“Our school was quarantined a couple of times. But whenever, you’ve done three lessons in person and … then you’re out for 14 days, you have to give them work. So, therefore, like the next two lessons now the in-person kids are doing them online also and they don’t quite understand how to do it online. … So, you’re not getting people doing their work. … And then you come back to school and you’ve got all these kids for two weeks that didn’t do their work online … and then you’re scrambling and everything and you get caught behind and … they didn’t get, they didn’t get it from the online, you’re 3 or more or lessons in and it’s something they should have learned 4 lessons back. And they were virtual at that time and they didn’t learn it because they didn’t do what they’re supposed to do. ... It was horrible.” (Teacher A) “Figuring out how to actually teach it in the gym, that was the biggest challenge, I guess, because, you know, we share a gym with the high school. There’s no like block off. So, we had to get their noise pollution and their light pollution. So, like, a PowerPoint, or like a projector really wasn’t an option. And trying to have them without us screaming, you know, … like yelling out the lessons to them and for them to be able to grasp it, that’s that was the biggest challenge.” (Teacher H) “Probably just the time constraint would be one of the big ones. And then obviously just being in and out of school.” (Teacher M) “I think when we did the online survey to start SIPsmartER, we had to stagger them because I’ve got the one Wi-Fi accessible point here in my classroom and there were twenty eight kids or so. … We all couldn’t log on so I had to stagger everybody to be able to do that. So the bandwidth is not great.” (Teacher E)
Student Engagement
	63% (n=12)
	• Perceived that students were less engaged during virtual delivery. (COVID-specific) • Difficulties promoting and assessing accountability during virtual instruction. (COVID-specific) • Kids prefer physical activity lessons to Kids SIPsmartER and other health lessons. (COVID-specific, general) • Physical education and health classes are often viewed as a lesser priority so less priority on work and students can be pulled for other subjects. (general, COVID-specific)	“… We do Kids SIPsmartER the first day of the week and it holds them accountable because they’re in person … You have to hold their feet to the fire or they’re not going to do it.” (Teacher L) “Maybe not the teaching aspect of it, but when we were fully virtual, actually getting them to complete, you know, the workbook, that that was the biggest challenge because, you know, they’re not in our class and we can’t stand over them and watch them, you know, do it and hold them accountable.” (Teacher H) “I think a little bit of it is because it’s you know, they’ve been so pent up. … I think they don’t want to have to sit and listen to something. They just want to be up moving around. (Teacher M) “And, you know, with gym, the kids get pulled for other classes.” (Teacher G)

Teachers and principals identified 6 specific benefits across 3 domains: perceptions of impact (1 benefit), conceptual fit (3 benefits), and materials and lesson design (2 benefits). All identified benefits reflected facets of Kids SIPsmartER that exist regardless of the COVID-19 context. Almost all (89%, n=17) teachers and principals identified benefits related to Kids SIPsmartER’s conceptual fit and to materials and lesson design. Approximately half (42%, n=8) identified the potential impact of the program or shared specific anecdotes about impacts.

Participants identified 13 specific barriers across 4 of the domains: conceptual fit (3 barriers), material and lesson design (2 barriers), program implementation (4 barriers), and student engagement (4 barriers). Of these barriers, those related to the domains of conceptual fit and material and lesson design reflected barriers that exist regardless of the COVID-19 context while those related to program implementation and student engagement were predominantly related to the COVID-19 context. Most teachers and principals identified barriers related to student engagement (63%, n=12) and almost half identified barriers related to program implementation (47%, n=9), yet only 26% (n=5) and 11% (n=2) identified barriers related to conceptual fit and material and lesson design, respectively.

DISCUSSION

Study findings provide descriptive evidence of how an on-going behavioral intervention can be strategically and pragmatically adapted to meet the needs of the enrolled schools. Specific to Kids SIPsmartER, findings suggest the adaptations to the implementation protocol for its classroom component supported program adoption, delivery, and satisfaction. All schools maintained the intervention, delivered all 12 lessons, completed training activities and fidelity checklists at a high rate, and reported multiple benefits of the program. Furthermore, findings from this study extend the existing literature relevant to adapting school-based nutrition education focused EBIs. ^2,8,9,15 Importantly, while these data were collected during the pandemic, the stressors faced by the participating schools reflect those commonly faced by schools, particularly those in health disparate communities [i.e., core tested subjects prioritized over physical education, health, and other non-SOL tested subjects; highly stressed students due to home and community context; teacher disconnection from students; and teachers feeling unstable due to frequent instructional changes (e.g., new curricula, changing school leadership)]. ⁷ This suggests that study implications are relevant for nutrition education focused EBIs, regardless of whether they are implemented during high-stress situations. Specifically, findings highlight 7 takeaways relevant to adapting and developing implementation protocols for school-based EBIs.

First, it is important to utilize existing frameworks – such as the FRAME, which was used in this study¹⁵ – to direct and report on the adaptation process. These frameworks provide important guidance that can help inform whether an adaptation needs to be made, what barriers it might impact, and if it has the potential to be fidelity consistent. Additionally, they provide structure from which adaptations to an EBI and its implementation protocol can be tracked and reported. ^14–16 Furthermore, using a systematic process to track and report adaptations allows for evidence about adaptations to be shared more completely, providing other researchers or interventionists with more complete descriptions that they can use to develop or change their EBIs. Building this evidence is necessary to support the wider dissemination and implementation of nutrition education focused school-based EBIs. Notably, while a few school-based interventions have used the FRAME to systematically describe adaptations, those that have used it have not been nutrition education interventions. ^31–34

Second, adaptation of an EBI can be facilitated by having a clearly delineated implementation protocol that incorporates activities that map to established implementation strategies, such as those identified by SISTER. ²⁶ Having a documented implementation protocol made it possible for the research team to identify which of Kids SIPsmartER’s implementation activities could be adapted and determine if new strategies needed to be developed. Without a set implementation strategy, the research team would not have been able to record and report on adaptations as completely. Also, if researchers and interventionists were to clearly describe their implementation protocols, it would not only support the execution of their specific EBI but also inform others about potential activities that “work.” Moreover, aligning specific activities with known implementation strategies can help establish this evidence more systematically.

Third, findings support the use of strategic adaptations, such as those that address specific needs or issues that impact specific program outcomes. Reasons for these adaptations will vary by the school-based EBIs implementation context, so researchers and interventionists must identify the unique needs of the schools with which they are working before making adaptations. In the current study, adaptations were made to reduce teacher burden and address COVID protocols as a means to maintain program delivery, teacher engagement, and program satisfaction. Furthermore, being strategic in planning for adaptations also means that the research team can ensure the adaptations promote relative consistency of implementation and limit major deviations from the core components. ⁹ For example, in this study, research team members adapted the 12 approximately 45-minute lessons into 10-to-20-minute videos. While this adaptation reduced the lesson time due to the inherent inability to have a class discussion during a video lesson, these videos addressed all lesson objectives and maintained key content and directions for worksheets. This helped reduce deviations in delivery because, if teachers had to adapt in-person lessons to virtual lessons, they may have focused on content they felt was the most important to include instead of adapting the whole lesson, which would have led to drift in content. Lastly, being strategic allows teachers to focus on educating and engaging with their students instead of developing resources; thus, removing unnecessary burdens.

Fourth, findings highlight making adaptations intended to make the EBI’s delivery easier for schools. How adaptations can reduce burden are influenced by the EBI’s design and its implementation context. The adaptations to the Kids SIPsmartER implementation protocol sought to make intervention delivery easier by reducing these burdens in multiple ways: providing resources compatible with the schools’ virtual instruction platforms (student facing videos, electronic workbooks), simplifying paperwork (electronic fidelity forms), and allowing the delivery timeline to work with the realities of the school (flexible delivery schedule).

Fifth, implementation approaches may vary across schools when adaptations are introduced, as schools will use those adaptations that best meet the needs of their school context. For example, in this study, schools engaged with the adapted student videos and electronic worksheets differently, which lead to different implementation approaches. One school delivered the program entirely virtually so that they could use in-person class time for physical activity. Two schools incorporated the student videos into their in-person classes by showing the videos to deliver content and pausing them to engage students in conversations. Importantly, this takeaway mirrors previous findings that allowing for opportunities for personalization and flexibility were identified by PE teachers as key for supporting the transition of their instruction to virtual during the COVID-19 pandemic. ⁶ Thus, as adaptations add flexibility to EBIs’ implementation, their incorporation can move implementation away from a “one-size-fits-all” approach. It is important for researchers and interventionists to be mindful of this so adaptations are made to implementation activities for which there can be differences among schools.

Sixth, while adaptations can alleviate barriers to implementation, they may not remove all barriers. For example, in the current study, over half of the teachers and principals noted student engagement, particularly during virtual learning, as a barrier. While some of the adaptations (e.g., student facing videos) were designed to help increase student access to content, they could not fully address the drivers that contributed to reduced student engagement. These drivers included systemic causes such as shifts in structure and routine and varying ability to access online learning, was outside the scope of how the research team, teachers, and principals could adapt the implementation protocol. It is important to note, student engagement was a well-documented barrier for schools across the country during the pandemic, especially among students who were learning virtually. ^35,36

Seventh, making decisions about adaptations to an EBI’s implementation protocol is less complicated for researchers and interventionists when data are not being collected as part of a research trial’s primary outcome assessment period. Adaptations can impact implementation fidelity and maintaining consistency in implementation is vital to understanding whether an EBI can impact targeted health and behavior outcomes. This consistency is crucial during collection of primary outcome data, which provides evidence of the intervention’s efficacy. In this study, all schools were beyond the phase of the trial in which primary outcome data were being collected. This limited concerns for how the adaptations would impact the evaluation of whether Kids SIPsmartER could impact its targeted behavior – sugar-sweetened beverage intake – and gave the research team more latitude to change the implementation protocol.

Study findings should be viewed within the context of several limitations. First, this study may have limited generalizability as it was conducted in 7 schools in 1 specific geographic region in the United States. Yet, looking across interviews code and meaning saturation were reached, ³⁷ as participating teachers and principals reported that their schools faced many of the same challenges to instruction. Notably, these challenges reflect those impacting schools across the country. ^7,35,36,38 Likewise, the benefits and barriers identified to implementing Kids SIPsmartER reflect those from studies in other communities during and prior to the pandemic. ^8,17 Second, it was not possible to completely understand the impact of the adaptations on fidelity, due to only approximately 50% of lessons being completed in-person and not being able to assess how students engaged in the on-line lessons. Third, it was beyond the scope of this article to report on the impact of these adaptations on the sugar-sweetened beverage behaviors of middle school students. Yet, these effectiveness analyses are planned, including comparing pre-COVID intervention effects to effects during the 2021–2022 school year. The mixed methods process evaluation process data presented in this paper will be critical in the interpretation of the effectiveness data, when available, and overall study conclusions of this type 1 hybrid trial. Despite these limitations, this study used an established framework and systematic coding approach to report adaptations and related process findings of an EBI among middle schools in a high-need, rural region in the United States. Furthermore, findings provide timely insight into a significant gap in the literature related to the implementation of nutrition education focused EBIs.

Utilizing an adapted implementation protocol supported the 7 schools in maintaining Kids SIPsmartER during the COVID impacted 2020–2021 school year and implementing all 12 lessons. However, schools varied in how they implemented the classroom curriculum based on the needs of the individual schools’ contexts. Yet, even in this turbulent school year, teachers and principals consistently identified multiple benefits to implementing the program during this school year. While barriers to implementing Kids SIPsmartER were identified, specific barriers were less consistently identified compared to benefits and were surmountable. These findings highlight the importance of allowing for strategic and pragmatic adaptations to nutrition education focused EBIs to support their execution in schools experiencing high stress situations.

IMPLICATIONS FOR RESEARCH AND PRACTICE

Findings from this study have implications for promoting the adoption and implementation of EBIs in schools. Study implications are relevant for the implementation of school-based EBIs when high internal and external stress exist that foster novel or exacerbate existing implementation barriers. This is because the systematic use of pragmatic adaptations allows for the consideration of the context and instructional flexibility needed to successfully adopt and deliver the program. Increasing the potential for nutrition education focused EBIs to be adopted, implemented, and sustained in high-stress schools, could contribute to reducing health disparities. However, to fully understand the implications of strategic and pragmatic adaptations to implementation protocols, effectiveness and reach outcomes of the EBI when implemented with the original and adapted implementation protocols must be compared. This is a next step to understanding the impact of these adaptations on Kids SIPsmartER’s outcomes. Furthermore, to support the wider integration of EBIs, particularly those addressing nutrition and other energy-balance related behaviors, into schools, researchers and interventionist would benefit from incorporating implementation strategy frameworks, like SISTER, into the design of the implementation protocols for their nutrition education EBIs and when adapting their implementation protocols use the FRAME or another frameworks to systematic guide and report on the adaptations.