Feasibility, Acceptability, and Preliminary Effectiveness of a Reflective Supervision Professional Development Series Within Early Care and Education

Abstract

The positive impacts of Reflective Supervision (RS) are becoming increasingly evident. This approach may be especially important for early childhood teachers, who must deliver academic curricula and attend to children’s social-emotional development while maintaining high-quality program standards in increasingly challenging environmental contexts. However, to effectively provide RS to early childhood teachers, supervisors must first be trained in this approach. This paper describes the results of one such innovative training effort using data from a statewide, community-based program evaluation and a pilot open trial in the United States. Participants (N = 83 supervisors) were offered foundational training in RS followed by monthly skill-building sessions. We used a concurrent mixed methods approach, integrating observational, survey, and interview data collected simultaneously to assess feasibility, acceptability, and preliminary effectiveness of the RS professional development series. Data integration was achieved through contiguous narrative and joint display approaches.

Research Findings:

The RS series was feasible to implement, acceptable to participants, and led to improved reflective supervisory competencies.

Practice or Policy:

The RS series may represent an opportunity to effectively support the early care and education workforce, and in turn, benefit children. We discuss some differences between samples that may be useful for future implementation efforts across early care and education settings.

Reflective Supervision (RS) is a favored supervisory approach in early childhood settings given its potential to increase reflective functioning in infant and early childhood professionals, promote more sensitive interactions between professionals and the children and families they serve, and ultimately foster healthy development in our youngest children (Brandt, 2014; Fenichel, 1992; Heffron & Murch, 2010; Heller & Gilkerson, 2009; Pawl, 1995). RS has been adopted—and is even mandated—by many U.S.-based programs determined to enhance the quality of the services they provide, including home visiting, early care and education (ECE), and community mental health organizations, among others (Eggbeer et al., 2010; Huffhines et al., 2023; Osofsky & Weatherston, 2016; Tobin et al., 2024). Over the past several decades, this approach has gained traction in early childhood spaces through theoretical and anecdotal evidence of its value (Eggbeer et al., 2007; Tobin et al., 2024), but research on RS has lagged behind. Now, however, empirical evidence for RS is also growing, and with it, the opportunity to deepen our knowledge of this widely used practice. Specifically, an emerging evidence base shows increased reflective capacity and improved well-being among professionals who received RS, and though less explored, greater use of responsive, supportive practices with children and families (see Huffhines et al., 2023 and Tobin et al., 2024 for reviews). Although most research suggests that receiving RS is beneficial, virtually no studies have examined how supervisors themselves become trained to deliver RS (see Low et al., 2018 for one exception). It is not enough to know that professionals benefit from RS if we do not also understand how supervisors are adequately trained to provide high-quality RS in the first place. Investigating the types of professional development in RS that supervisors receive, whether supervisors are able to engage with and make use of such training, and whether they then experience positive change in their supervisory practices will be critical for increasing our understanding of how we can get RS into the bricks of early childhood settings (Heffron & Murch, 2010). As such, the current paper describes a professional development series in RS and uses multiple data sources from a statewide, community-based program evaluation and a pilot open trial in the United States to assess the feasibility, acceptability, and preliminary effectiveness of this training series for ECE supervisors. This work represents a first step towards understanding how to effectively train supervisors to implement RS within ECE settings.

Defining Reflective Supervision

RS is a distinct supervisory approach informed by an infant mental health perspective which has, at its core, the belief that all learning occurs within the context of nurturing relationships—not only for children, but for adults too (Brandt, 2014). RS is therefore a collaborative, reflective relationship between a supervisor and supervisee, wherein the supervisee regularly talks with their supervisor about their work with young children and families, with the goal of receiving support and strengthening their professional practice (Heffron & Murch, 2010; Heller & Gilkerson, 2009; Silver et al., in press; Watson et al., 2016;). Within this relationship, the supervisee is invited to safely explore the range of emotions and reactions that are evoked while doing the difficult work of caring for children and families, and also to broaden their thinking through perspective taking and collaborative problem solving (Heffron & Murch, 2010; Heller & Gilkerson, 2009; Silver et al., in press; Watson et al., 2016;). Over time, RS is thought to build greater awareness of the emotional experiences of the self and others and the parallel processes taking place across relationships, while also supporting the skill development needed to engage in positive, evidence-based practices; the result is a professional who is more attuned to their own and others’ needs and responses, and is more sensitive in their delivery of the gold standard practices and curricula for their field (Heffron & Murch, 2010; Heller & Gilkerson, 2009; Silver et al., in press; Watson et al., 2016;).

Within a RS relationship, the supervisor and supervisee each hold specific roles (Heffron & Murch, 2010; Heller & Gilkerson, 2009). Supervisors are responsible for providing RS on a relatively predictable schedule in order to build a consistent, trusting relationship with the supervisee over time, and to establish a regular pattern of discussing the supervisee’s experiences with their work, rather than only meeting in response to a crisis or for evaluative purposes (Heffron & Murch, 2010; Heller & Gilkerson, 2009). Supervisors are also tasked with supporting supervisees in expanding their reflective functioning, and they do this using a variety of skills, including asking open-ended questions to encourage exploration of one’s own and others’ identities, feelings, and behaviors, and helping the supervisee to slow down and consider their work from multiple perspectives, broadening their understanding of a situation and informing next steps (Heffron & Murch, 2010; Heller & Gilkerson, 2009). Supervisors facilitate decision-making and problem-solving in collaboration with the supervisee, so that actions are mutually determined by both parties and supervisees are participating fully in their own professional growth (Heffron & Murch, 2010; Heller & Gilkerson, 2009). Supervisees also hold responsibilities within the RS relationship, in that they should come to supervision prepared with topics they would like to explore, be willing to openly discuss their experiences, feelings, and concerns, and should follow through on collaboratively determined plans (Heffron & Murch, 2010; Heller & Gilkerson, 2009). The resulting relationship, sometimes termed the “reflective alliance,” is a jointly created partnership defined by safety, trust, and authenticity, that necessarily changes over time as the dyad comes to know each other (Watson et al., 2016). Importantly, RS may look somewhat different across early childhood contexts (e.g., it may be difficult for supervisors and supervisees in ECE to meet as regularly as supervisory dyads in mental health settings, which traditionally build in time for supervision). In summary, RS is a dynamic, multifaceted practice designed to support a range of early childhood professionals, and has been described in great detail by multiple sources within the infant and early childhood mental health realm (e.g., Barron et al., 2022; Meuwissen et al., 2022; Shea et al., 2022; Silver et al., in press).

Reflective Supervision in Early Care and Education

In ECE settings, teachers must implement academic curricula alongside social-emotional curricula to encourage healthy, well-rounded development, and are also tasked with responding to difficult behaviors, communicating with caregivers, and creating a warm, nurturing, and safe classroom environment for all children (Bierman et al., 2017). Unfortunately, research has shown that teacher use of positive, evidence-based practices and curricula decreases significantly over the course of a school year, and that the biggest predictor of this phenomenon is lack of high-quality supervision (Hunter & Bierman, 2021; Pianta & Hamre, 2020; Trivette et al., 2014). Teaching effectively became even more challenging with the onset of the COVID-19 pandemic, as teachers now navigate increased child behavioral problems and learning delays, and attempt to support families managing numerous stressors, all within the context of staffing shortages and other systemic challenges (Bierman et al., 2023; Egan et al., 2021). It is unsurprising, then, that teachers express feeling stressed, overwhelmed, and burned out (Quinn et al., 2022). RS may provide teachers with the support they need and deserve, and in turn, may improve the quality of ECE programs for young children.

Several studies have illustrated how RS may benefit teachers in ECE, though with fewer than a dozen studies conducted to date, these findings are not conclusive (see Huffhines et al., 2023 for a review of strengths and weaknesses in existing RS research). For example, participants have reported that RS (or the related practice of reflective consultation) reduces teacher stress immediately through emotional containment, validation, and affirmation, and in the long-term by helping to clarify professional role, increase perspective taking, challenge one’s biases, and reorient towards self-efficacy; participants also reported reduced burnout and vicarious trauma, greater engagement with work, increased motivation for professional development, and skill growth, such as improved emotion regulation and increased insightfulness and reflective capacity (Barron et al., 2022; Harrison, 2016; Hazen et al., 2020; Shea et al., 2022; Susman-Stillman et al., 2020; Virmani & Ontai, 2010; Watson et al., 2014). These benefits may reverberate outward to families, as participants reported feeling more competent talking with families about difficult things, and more capable of developing positive relationships with families overall (Barron et al., 2022). Successfully implementing RS in ECE programs may hold great promise for the workforce, and the children and families they serve, but it is still unclear what professional development is needed to support those in supervisory roles in using RS with their staff.

The Impact of Training Supervisors to Provide Reflective Supervision

A teacher cannot receive high-quality RS unless their supervisor is trained to provide it. However, very few studies have evaluated professional development opportunities for supervisors. A study in the field of maternal and infant/early childhood home visiting is the exception; Low and colleagues (2018) found that training home visiting supervisors in RS resulted in improvements in reflective supervisory competencies on two self-report measures. Yet, more research must be done to understand what types of professional development trainings in RS are effective in building supervisory capacity, and how they translate to different settings. In ECE, it will be important to determine whether professional development focused on RS is feasible for supervisors to complete amidst their other job responsibilities, whether supervisors like and want training in RS, and whether training in RS works. In answering these questions, we will begin to establish a pathway for becoming a reflective supervisor in the ECE field (admittedly a lifelong process, though we view initial training as a critical first step). Cultivating this supervisory workforce may, in turn, allow more widespread and successful implementation of RS where it is needed—in the places where children learn. Thus, the purpose of this paper is to describe one approach to training ECE supervisors in RS, and to answer the following questions using several data sources in multiple ECE settings: was the RS series (a) feasible, we hypothesized that supervisors will meet attendance thresholds (≥ 70% attendance, ≤ 20% attrition), (b) acceptable, we hypothesized that supervisors will be satisfied with the series, and (c) effective, we hypothesized that supervisors will report positive change in their reflective supervisory competencies.

Material and Methods

Overview

To determine feasibility, acceptability, and preliminary effectiveness of the Rhode Island Reflective Supervision Series within ECE, we analyzed data from two samples within this state. Integrating data from multiple samples allowed this mixed methods study to achieve a higher level of rigor (Fetters et al., 2013). Given our objective of assessing whether this professional development series is applicable and appropriate for a range of ECE settings, we utilized data collected from supervisors overseeing multiple types of classrooms representative of the Rhode Island ECE landscape (e.g., state-funded pre-K, Head Start, private for-profit, private non-profit) in the context of a community-based program evaluation (i.e., Sample A), and supervisors overseeing state pre-K classrooms in the context of a pilot open trial (i.e., Sample B). Although overseeing state pre-K classrooms was required for participating in the open trial, in Rhode Island, it is common for early learning centers to house more than one type of classroom. Thus, supervisors participating in the open trial often supervised teachers in other types of classrooms (e.g., Head Start, private for-profit, private non-profit) as well. Importantly, the difference between samples is largely contextual (i.e., one is a community-based program evaluation while the other is a research study), as the participants in both samples are demographically similar and supervise the same types of classrooms in one small state. Furthermore, all participants received the same curricula and structure of training in RS, described below. It should be noted that ECE supervisors in our state do not typically receive formal training in RS, and there is not currently a pathway for doing so outside of the training detailed in this paper.

The Rhode Island Reflective Supervision Series

The RS series for ECE professionals included two curricula (Foundations of RS and RS: Skill Development Workshop [i.e., Workshop]) that have been described in detail in Silver et al. (in press). These curricula were developed using best practice guidelines for building competency in RS (e.g., Alliance for the Advancement of Infant Mental Health, 2018; Heffron & Murch, 2010; Heller & Gilkerson, 2009), incorporate infant and early childhood mental health concepts (e.g., parallel process), introduce a framework for diversity-informed and anti-bias and anti-racist practices, and attend to diverse learning styles by balancing content (e.g., didactics, readings) with experiential practices (e.g., modeling, role-play of skills, video-based examples). The curricula were delivered by one or more trainers who held a doctorate in clinical psychology or clinical child psychology, and had achieved infant mental health endorsement through the state’s infant mental health association. For Sample A, multiple trainers who were endorsed or endorsement eligible were present, while Sample B had one endorsed trainer. Although trainers were all endorsed psychologists, the curricula can be delivered by trainers with different educational backgrounds. Furthermore, the RS series can be delivered in-person or virtually, allowing for flexibility based on public health guidelines or preference. Delivering the series virtually requires only minor adaptations (e.g., use of breakout rooms rather than in-person discussions); these adaptations are described in Silver et al. (in press). Sample A participated in a blend of in-person and virtual offerings in 2019, 2022, and 2023, while Sample B participated solely in-person over the 2022–2023 academic year.

Supervisors first completed Foundations of RS, an eight-hour training that defines RS, describes what it means to embody a reflective stance, and focuses on bringing RS skills and strategies to life so that participants can begin to use them in their work. Upon completing Foundations of RS, supervisors were eligible to participate in RS: Skill Development Workshop (i.e., Workshop), a monthly small-group experience designed to extend the learning gained in Foundations of RS and support supervisors as they implement RS within their programs. Each monthly session included a supportive check-in with participants, facilitated discussion of that month’s reading on RS, and a group-based reflecting team experience. The Workshop took place over six months, resulting in six sessions. The six sessions for Sample A were each an hour and a half, while the six sessions for Sample B were two hours. The extended time for Sample B allowed for administration of research measures; the content both samples received was the same. Thus, supervisors in both samples were offered a total of 17 hours of professional development content focused on RS.

Participants

Across two distinct samples, participants are representative of the range of ECE classrooms in Rhode Island, including state-funded pre-K, Head Start, private for-profit, and private non-profit classrooms; family childcare was not represented. Because the RS series has been funded in numerous ways, eligibility requirements for participation have varied across time.

Sample A (Community-Based Program Evaluation)

Sample A included 70 participants who were part of a community-based program evaluation. For the first offering of Foundations of RS in 2019, which was an in-person pilot of the curriculum within ECE, any ECE supervisor in the state was eligible to participate. For the subsequent virtual offerings beginning in 2022, there were minor adaptations in structure and timing, but not substantive content, and all ECE supervisors in the state, except those solely overseeing state pre-K classrooms, were eligible. Eligible supervisors received outreach about the RS series via email and/or through hard copy mail. Supervisors in this sample enrolled in Foundations of RS as a stand-alone experience. An invitation to participate in the optional Workshop was later sent to participants who had completed Foundations of RS, beginning in 2022. See Table 1 for sample demographics.

Table 1.

Demographics of Sample A & Sample B

	Sample A (N = 64; 6 missing)		Sample B (N = 12; 1 missing)
	M (SD)	Range	Mean (SD)	Range
Age	48.7 (11.8)	27–72	40.3 (9.1)	27–56
	N	%	N	%
Gender
Female	63	98%	11	92%
Male	1	2%	1	8%
Other	−−	−−	−−	−−
Race
White	60	94%	12	100%
Black	2	3%	−−	−−
Not reported	2	3%	−−	−−
Ethnicity
Hispanic/Latino(a)	4	6%	1	8%
Not Hispanic/Latino(a)	47	73%	11	92%
Not reported	13	20%	−−	−−
Education
High school diploma	1	2%	−−	−−
Some college	1	2%	−−	−−
Associate’s degree	4	6%	−−	−−
Bachelor’s degree	36	56%	6	50%
Master’s degree	19	30%	5	42%
Doctoral degree	1	2%	−−	−−
Other type of certification	1	2%	1	8%
Time spent in ECE field
1–3 years	1	2%	−−	−−
3–5 years	2	3%	1	8%
5–10 years	12	19%	2	17%
10 years or more	49	77%	9	75%
Time spent in current program
0–6 months	6	9%	1	8%
6–11 months	2	3%	−−	−−
1–3 years	11	17%	2	17%
3–5 years	6	9%	1	8%
5–10 years	16	25%	3	25%
10 years or more	23	36%	5	42%
Time spent as a supervisor
0–6 months	8	13%	2	17%
6–11 months	2	3%	1	8%
1–3 years	17	27%	−−	−−
3–5 years	14	22%	5	42%
5–10 years	9	14%	1	8%
10 years or more	14	22%	3	25%
Endorsement
Yes	−−	−−	−−	−−
No	64	100%	12	100%

Sample B (Pilot Open Trial)

Sample B included 13 participants who were part of a formative, pilot open trial designed to assess feasibility, acceptability, and preliminary indicators of effectiveness of the RS series in state pre-K and to identify contextual factors associated with implementation of RS in programs prior to a planned randomized controlled trial (RCT). As such, the expectation was that participants would complete both curricula, Foundations of RS and the Workshop. This was different from Sample A, wherein participants could choose to enroll in Foundations of RS without enrolling in the Workshop. Directors and education coordinators who supervised teachers working in state pre-K classrooms were eligible to participate. Supervisors were recruited at a state-wide meeting for education coordinators and through email announcements sent to all state pre-K supervisors. See Table 1 for sample demographics.

Procedures

Sample A (Community-Based Program Evaluation)

These data were collected in the context of program evaluation, thus we received Lifespan Institutional Review Board (IRB) approval for record review. The goal of our program evaluation efforts was to ascertain whether there were indicators of feasibility, acceptability, and effectiveness of the RS series to share with our funders, while minimizing burden for participants. During Foundations of RS, participants completed a demographics questionnaire, two questionnaires assessing their reflective supervisory competencies, and a satisfaction questionnaire. For the Workshop, participants repeated the questionnaires assessing their supervisory competencies at the first and last sessions and also completed a satisfaction questionnaire at the last session.

Sample B (Pilot Open Trial)

All study procedures were approved by the Lifespan IRB. Further, this study was pre-registered via the Open Science Framework Registries and can be viewed at https://osf.io/hu8kw. Results from the open trial pertaining to implementation factors, as well as indicators of effectiveness assessed via coding of audio-recorded supervision sessions, will be presented in forthcoming papers, given the large scope of this work (see Levitt et al., 2018 for rationale). Participants provided informed consent prior to any study involvement. Towards the beginning of Foundations of RS, participants completed baseline assessments, including a demographics questionnaire and two measures of reflective supervisory competencies. Participants repeated the questionnaires assessing their supervisory competencies at the first and last sessions of the Workshop, and following the last session, participants completed an individual interview which included questions related to feasibility, acceptability, and effectiveness of the RS series, as well as implementation-focused questions, which are not presented here. Participants received monetary compensation for their time completing study assessments.

Measures

Feasibility

Prior to this study, we determined that a 70% attendance rate and a 20% attrition rate were acceptable cut-points to demonstrate feasibility, per previous research. For both samples, feasibility was assessed using the following indicators: (1) percentage of registered participants who attended Foundations of RS (target ≥ 70%), (2) attrition from Foundations of RS (target ≤ 20%), (3) percentage of registered participants who attended the Workshop (target ≥ 70%), and (4) attrition from the Workshop (target ≤ 20%). For Sample B, feasibility was also assessed qualitatively, through individual interviews, which are described in more detail below.

Acceptability

Sample A.

For the program evaluation, we developed two questionnaires to determine acceptability/satisfaction with the RS series. The first questionnaire pertains to Foundations of RS, and asks participants to rate the training content, training techniques and activities, delivery of content, and facilitation skills via 13 items on a 5-point scale (“unsatisfactory” to “outstanding”). Satisfaction was defined as a rating of 4 or 5 in each domain. The questionnaire also includes three open-ended questions asking participants what they liked most about the training, what trainers can do to improve the training, and additional comments. The second questionnaire pertains to the Workshop, and asks participants to rate whether the sessions, the readings, and the reflecting team approach seemed applicable to their work and supported their professional development. These three items were measured on a 5-point scale (“strongly disagree” to “strongly agree”). Satisfaction was defined as a rating of 4 or 5 in each domain. Participants also responded to six open-ended questions asking them to describe what the Workshop has meant to them, what they have learned/how they have grown, what they found most helpful, what they found least helpful, what changes they would like to see, and additional comments. Qualitative responses to open-ended questions were inductively coded for themes by the first author (LH) and audited by research team members. Quantitative and qualitative responses from the questionnaires were used to descriptively assess acceptability.

Sample B.

For the open trial, acceptability of the RS series was assessed qualitatively, through semi-structured individual interviews with all supervisors who participated in Foundations of RS and the Workshop (N = 11 total; we sought a sample size of 10 to reach saturation for the constructs of interest). The interviews were also designed to identify contextual factors related to implementation of RS in programs, and thus an interview guide was developed using the Consolidated Framework for Implementation Research (CFIR; Damschroder et al., 2009). The interview guide was reviewed by multiple research team members, including three with qualitative expertise, and tested internally prior to use. The interview questions related to the current aim of assessing feasibility, acceptability, and preliminary effectiveness are presented in Table 2. If participants shared information about any of these three constructs at other times during their interview, that information was also included in qualitative analyses. The interviews ranged from approximately 30 minutes to an hour, were audio recorded, and were conducted by an experienced clinical interviewer who had not otherwise contributed to the study and was unknown to participants. All interviews took place within approximately a month of each other. The principal investigator of the study (LH) listened to the interviews and provided initial feedback to the interviewer following the first two interviews, and provided ongoing consultation as needed throughout the data collection period. Each interview was automatically transcribed via videoconferencing software, and then carefully reviewed by a clinical research assistant (IR) who listened to the recordings, edited the transcripts, and ensured that they were transcribed verbatim.

Table 2.

Interview Questions Related to Feasibility, Acceptability, and Effectiveness for Sample B

Construct	Interview Question	Probes (as needed)
Acceptability	What did you think about the foundational training and skill development workshops?	• What did you like or find most useful? • What did you dislike or find least useful? • What would you change?
Acceptability/Effectiveness	In what ways, if any, were the foundational training and skill development workshops helpful in supporting you, professionally or personally?	• Can you provide an example? • [If not helpful], why not?
Acceptability/Effectiveness	What other outcomes, if any, do you think were impacted by your participating in the foundational training and skill development workshops?	• Can you provide an example?
Feasibility	Is the time commitment for the foundational training and skill development workshops doable?	• [If yes], how did you make the time commitment work? • [If no], why not?
Effectiveness	If you were to describe to someone (e.g., a work colleague, a friend, a family member) what reflective supervision means to you, what would you say?	N/A
Effectiveness	From your perspective, what are the core characteristics of reflective supervision?	N/A
General	Is there anything else that we need to know about the foundational training or skill development workshops?	N/A

Effectiveness

For both samples, effectiveness was assessed using two questionnaires measuring participants’ perceptions of their own reflective supervisory competencies, the Reflective Supervision Rating Scale-Adapted (RSRS-A) and the Supervisor Rating Scale-Adapted (SRS-A). Participants completed both measures during Foundations of RS, and at the first and last sessions of the Workshop. To obtain an adequate sample size for quantitative analyses, data from Sample A and Sample B were combined.

The RSRS-A was adapted from the Reflective Supervision Rating Scale (Ash, 2010) to be a self-report measure rather than supervisee-report measure. The RSRS-A asks participants to assess their competencies as a reflective supervisor via 17 items on a 4-point scale (“rarely” to “almost always”). Example items include: “I have formed trusting relationships with my supervisees” and “I allow my supervisees time to come to their own solutions during supervision.” Items are averaged to derive a total score, which was used for analyses. The RSRS-A demonstrated adequate internal reliability in the combined sample at all three time points (α’s ranged from .82 to .90). The SRS-A was similarly adapted from the Supervisor Rating Scale (Dickstein & Wilson, 2008) to be a self-report measure. The SRS-A also invites participants to assess their competencies as a reflective supervisor, and does so via 14 items on a 5-point scale (“strongly disagree” to “strongly agree”). Example items include: “As a supervisor, I generally provide regularly scheduled supervision” and “As a supervisor, I help my staff to understand their strengths and limitations.” Items are averaged to derive a total score, which was used for analyses. The SRS-A demonstrated adequate internal reliability in the combined sample at all three time points (α’s ranged from .77 to .89). Means and standard deviations for both measures at all time points in the combined sample are presented in Table 3. For Sample A, effectiveness was not assessed qualitatively, given that participants’ written responses on the satisfaction questionnaires were necessarily brief, and did not provide rich enough data to make interpretations regarding participants’ understanding of curricula and use of supervisory skills. For Sample B, effectiveness was assessed qualitatively, through the individual interviews.

Table 3.

Descriptive Statistics and Bivariate Correlations for the Combined Sample

	1	2	3	4	5	6	7	8	9	10	11
1. RSRS-A 1	--
2. RSRS-A 2	.57*	--
3. RSRS-A 3	.15	.24	--
4. SRS-A 1	.56*	.41	.49	--
5. SRS-A 2	.39	.74***	.37	.64**	−−
6. SRS-A 3	−.10	.13	.76***	.47*	.41	−−
7. Age	.16	.21	.05	.22	.38	.20	−−
8. Education	.17	.32	−.09	.31	.53*	.11	.11	−−
9. Sup. Exp.	.09	.30	.33	.01	.20	.11	.32	.14	−−
10. ECE Exp.	.11	.35	.14	−−.01	.42	−−.14	.39	.03	.44	−−
11. Dosage	−.03	−.07	−.28	−.11	−.03	−.01	−.19	−.10	−.18	−.23	−−
Mean	2.94	2.74	3.33	4.19	4.10	4.43	45.33	−−	−−	−−	4.89
SD	.41	.36	.43	.42	.35	.37	10.28	−−	−−	−−	1.33
Minimum	2.18	2.18	2.53	3.43	3.50	3.79	27	−−	−−	−−	2
Maximum	3.69	3.71	4	4.86	4.79	5	65	−−	−−	−−	6

Note. RSRS-A = Reflective Supervision Rating Scale-Adapted. SRS-A = Supervisor Rating Scale-Adapted. Sup. Exp. = Years of experience as a supervisor. ECE Exp. = Years of experience in the early care and education field. 1 = Foundations of RS baseline assessment. 2 = First Workshop session. 3 = Last Workshop session.

^***p < .001

^**p < .01

^*p < .05.

Data Analysis

Quantitative Data

Change in perceived reflective supervisory competencies over time was examined quantitatively to determine effectiveness of the RS series in the combined sample. SPSS Version 28 was used to examine descriptive statistics, correlations, and potential covariates, as well as to conduct model testing. A repeated measures ANOVA with a Greenhouse-Geisser correction was used to examine change in supervisors’ perceived supervisory competencies, for both measures (i.e., the RSRS-A and SRS-A). Post hoc analyses with a Bonferroni adjustment were used to further understand any significant within-subjects effects.

Missing Data and Power Analysis.

Data from three time points (Foundations of RS, first and last sessions of the Workshop) were available from 15 total participants (4 from Sample A, 11 from Sample B) for the RSRS-A, and 19 total participants (8 from Sample A, 11 from Sample B) for the SRS-A. Sample B had no missing data. As shown in Figure 1, 70 participants from Sample A attended Foundations of RS, 11 attended the Workshop, and 10 completed the Workshop, hence the smaller pool of participants from Sample A. Further, four participants from Sample A were missing RSRS-A data from the last Workshop session due to a systematic REDCap data collection error for that cohort. Two participants from Sample A did not attend the last Workshop session, and therefore did not complete the program evaluation, resulting in missing data for both measures. Additionally, one participant from Sample A completed the Workshop twice, thus only the first set of responses was used.

Figure 1. Flow diagram depicting attendance and completion of the RS series in Sample A.

An a priori power analysis was conducted using G*Power version 3.1.9.7 (Faul et al., 2007) to determine the minimum sample size required to test change in supervisors’ perceived supervisory competencies over time. Results indicated the required sample size to achieve 80% power for detecting a small-to-medium effect (d = .25), at a significance criterion of α = .05, was N = 17 for a repeated measures ANOVA. These calculations suggest that our study is adequately powered to test change across three time points for the SRS-A (N = 18), but is slightly underpowered for the RSRS-A for this effect size (N = 15).

Qualitative Data

Transcripts were coded using a directed content analysis approach (Hsieh & Shannon, 2005), wherein coders applied codes from a CFIR-based coding manual (Damschroder et al., 2022). In order to capture a full range of participant experiences, emergent codes that were not covered by the CFIR were generated and added to the codebook as needed; however, no new codes related to feasibility, acceptability, or effectiveness emerged. All transcripts were coded by two research team members, the principal investigator (LH) and the clinical research assistant (IR). These two coders received training and directed readings from a senior team member with extensive expertise in qualitative methods (ARE). The lead coder (LH) had also received previous training in qualitative methods through coursework and workshops. The two coders completed consensus coding for all transcripts, where they resolved questions and coding discrepancies, and defined emerging codes. Coder agreement was excellent (Kappa = 0.90). The coders also regularly met with the senior team member for consultation, which included discussion of the coders’ questions, coding decisions, and the coders’ alignment with CFIR construct definitions. After coding the first transcript, coders immediately received consultation; this process was repeated for the second transcript as well. Data analysis was supported by Dedoose qualitative software.

To maintain rigor and trustworthiness, we employed several strategies over the course of the study. Prior to coding, the coders reflected on potential sources of bias (e.g., how engaging in RS themselves has informed their view of the approach); the principal investigator and lead coder (LH) also wrote a positionality statement which can be viewed at https://osf.io/hu8kw. During the coding process, the coding team used written memos to observe and reflect on bias, and discussed potential or actual bias during consensus and consultation meetings. The coders also kept a logbook of the discussion, and coding decisions made at each consensus or consultation meeting. Additionally, the coders sought secondary input when needed from team members who were familiar with RS and ECE, but were not immersed in coding and data analysis. Finally, this study used a triangulation design to increase validity by corroborating multiple findings (Creswell & Plano Clark, 2011; Greene et al., 1989). For instance, we used more than one data source (i.e., observational, survey, interview), included participants from different ECE settings, and employed two coders.

Data Integration

In our concurrent mixed methods approach, we synthesized data at the analysis stage from several sources and across samples, to comprehensively understand how participants perceived, and benefited from, the RS series. Observational (i.e., attendance and attrition), quantitative (i.e., survey), and qualitative (i.e., written response and interview) data were collected in parallel, and analysis for integration of these data began after all data collection was completed (Fetters et al., 2013). Following Journal Article Reporting Standards (JARS) for Mixed Methods Research (Levitt et al., 2018), we analyzed our different types of data separately and then merged these data to present a convergent, integrated overview (Fetters et al., 2013). We achieved data integration through connecting and merging methods. First, we used a connecting method, wherein our databases were linked through sampling. Specifically, all participants contributed all types of data, with some differences between samples (e.g., Sample A provided qualitative data through written responses to survey items, while Sample B participated in interviews). Second, we used a merging method, wherein the databases were brought together for comparison, and data was integrated through a contiguous narrative approach and a joint display approach.

Results

Feasibility

Sample A (Community-Based Program Evaluation)

Observational Results.

Attendance at Foundations of RS and the Workshop met the target attendance rate of ≥ 70% (see Figure 1 for flow diagram). Attrition from Foundations of RS exceeded the target of ≤ 20%. Attrition from the Workshop, however, was less than 20%. Attendance by session is presented in Table 4.

Table 4.

Workshop Sessions Attendance Rate in Sample A

Session	Attendance Rate
1	100%
2	91%
3	73%
4	73%
5	73%
6	64%
Average attendance across sessions	79%
Individual attendance rates (N=11): 5 participants attended 100% of sessions, 2 attended 83%, 1 attended 67%, 2 attended 50%, 1 attended 33%

Note. Reasons for absences were not collected as part of the program evaluation.

Sample B (Pilot Open Trial)

Observational Results.

Attendance at Foundations of RS and the Workshop met the target attendance rate of ≥ 70% (see Figure 2 for flow diagram). There was no attrition from Foundations of RS or the Workshop, thereby meeting the target attrition rate of ≤ 20%. Attendance by session and reasons for absences are presented in Table 5.

Figure 2. Flow diagram depicting attendance and completion of the RS series in Sample B.

Table 5.

Workshop Sessions Attendance Rate and Reasons for Absences in Study B

Session	Attendance Rate	Number of Absent Participants with Reason
1	91%	1: Work conflict (conference)
2	73%	1: Work conflict (meeting) 1: Work conflict (training) 1: Personal conflict (illness)
3	91%	1: Work conflict (training)
4	64%	4: Personal conflict (illness)
5	64%	2: Work conflict (conference) 1: Work conflict (meeting) 1: Personal conflict (vacation)
6	91%	1: Work conflict (meeting)
Average attendance across sessions	79%
Individual attendance rates (N=11): 4 participants attended 100% of sessions, 3 attended 83%, 1 attended 67%, 3 attended 50%

Qualitative Results.

Feasibility for the RS series was further demonstrated through qualitative data. All participants expressed that the time commitment to participate in the RS series was feasible, though some participants were initially apprehensive, as exemplified by this quote: “[The RS series] was doable. At first, I was like, ‘oh man, I don’t know if I can commit to this.’ The first [Workshop session] was like, all right, okay. I need to do this. I need to get there on time. But then it became a routine.” (Sample B Participant 11 [B11]). Participants also largely agreed that meeting once per month for the Workshop sessions was the right frequency: “I think personally, [the time commitment] was doable, but I also was really invested in making it a priority for my growth this year. I set the tone at the program, saying, this is an important piece for me to make sure I attend. It’s not negotiable kind of. I think, any more than once a month might have been hard to get out of the program.” [B2]. However, a few participants wished for more frequent meetings, while noting that this would be a challenge for most supervisors in ECE: “It would have been nice to meet more than once a month, but I know that had that been the case, it would have been just another thing on top of everything else that everyone is doing so would have been tricky to do. Because, I mean, I found it so useful that I would have loved to have more of those sessions.” [B9]. Although every supervisor interviewed stated that they believed the time commitment for participation in the RS series was doable, the three supervisors who missed three Workshop sessions shared what got in the way for them. This is captured by one quote in particular: “[We] wear very many hats so it was really hard, even though it was on the calendar, and it was a commitment already known by my supervisor. If something happened in my center I’d have to cover, or again if I was sick, I wouldn’t be able to go.” [B10].

Acceptability

Sample A (Community-Based Program Evaluation)

Quantitative Results.

Quantitative results suggested that participants were highly satisfied with Foundations of RS. Ratings from the 55 participants who completed the satisfaction questionnaire following Foundations of RS indicated that 91% of participants were satisfied with the training content, 95% with the training techniques and activities, 88% with the delivery of content, and 90% with the facilitators’ skills. Participants were also highly satisfied with the Workshop. Ratings from the 8 participants who completed the satisfaction questionnaire following the last Workshop session indicated that 100% of participants were satisfied with the Workshop sessions, 75% were satisfied with the readings, and 88% were satisfied with the reflecting team.

Qualitative Results.

Qualitative results mirrored these findings. Participants expressed a high degree of enthusiasm for Foundations of RS in their written responses, for example: “It was a great series and I look forward to continuing my journey into the more practical aspects of RS.” (Sample A Participant 6 [A6]). We observed four overall themes that emerged in the satisfaction questionnaire for Foundations of RS, demonstrating that participants enjoyed: (1) increasing their knowledge of RS [Knowledge], (2) learning specific, relevant strategies [Strategies], (3) connecting with other participants [Connection], and (4) feeling safe and heard [Safety]. Regarding the first theme, Knowledge, participants noted that they received information about RS in a clear and understandable way, allowing them to define RS and understand what it means to provide supervision in a reflective manner, as well as get an overall sense of how RS can be applied in their setting. For example, one participant described the applicability of what they learned about RS, “[Foundations of RS] brought information together in a way that made practical sense. This is something that I will be able to use on a regular basis with my staff.” [A7]. Regarding the second theme, Strategies, participants described specific skills and strategies that were relevant to them. For example, a participate wrote, “I enjoyed… learning how to ask the right questions and expand on those questions.” [A1]. Regarding the third theme, Connection, participants valued opportunities for connection with other participants through discussion. One participant shared, “I especially liked hearing everyone’s different perspectives on how they handle certain situations.” [A2]. Regarding the fourth theme, Safety, participants brought up how they appreciated feeling safe and heard during the training, which allowed them to participate more fully. For example, a participant stated, “Trainers acknowledged and made me feel comfortable when I participated.” [A8]. Taken together, supervising professionals who have participated in Foundations of RS have not only appreciated the content of the training, but also the process.

Participants similarly expressed high levels of satisfaction with the Workshop in their written responses. We observed three overall themes that emerged in the satisfaction questionnaire about the Workshop sessions, demonstrating that participants enjoyed: (1) connecting with other supervisors [Enhancing Connection], (2) expanding skills through practice [Expanding Skills], and (3) implementing skills and strategies within programs [Using Skills]. First, regarding the theme of Enhancing Connection, participants greatly valued being able to share their experiences while also learning from other supervisors and receiving support in the challenging work they do each day. Participants made many statements about how much they enjoyed connecting with other supervisors, such as: “[The job] is a bit isolating at times and [it’s] wonderful to hear other’s perspectives.” [A5]. Second, regarding the theme of Expanding Skills, participants described how they were expanding their skills in RS through practice in the Workshop sessions, and in several responses, it was evident that participants were growing more comfortable with specific skills: “These monthly meetings have given me hands-on opportunities to improve my supervision skills and build my confidence as a supervisor through practice,” [A9] and “I have gotten better at pausing, slowing down a conversation, and getting curious. It has become easier to ask questions instead of giving advice as my first response.” [A3]. Third, regarding the theme of Using Skills, because of participation in the Workshop sessions participants described being better able to implement skills and strategies related to RS within their programs: “the [reflecting teams] – they really helped me understand the way of implementing [RS] in my center,” [A4] and “I have learned that RS actually makes my job easier. It fosters positive relationships with my staff, reduces conflict with staff, boosts morale, and helps staff improve their practice as well.” [A9]. These qualitative results indicated that people were not only highly satisfied with the Workshop content, but again enjoyed the experience of being with other supervisors, while also actively implementing what they were learning in their programs. Participants also provided feedback around specific changes they would like to see in the RS series, though these were individual, and no common theme emerged across two or more participants.

Sample B (Pilot Open Trial)

Qualitative Results.

Participants were extremely satisfied with the RS series as whole, per their responses in individual interviews. All 11 participants reported that they found the RS series acceptable. This is exemplified in a quote from one participant: “I think [the RS series] is just such a necessary component that we need in education, and what we need to support one another. And if anybody in Rhode Island pre-K or wherever this is offered, would ever ask me, ‘Do you recommend?’ I would highly recommend this training as I just found it really essential to what I do every single day.” [B6]. We found three overall themes pertaining to what participants appreciated most about the RS series: (1) increasing their knowledge of RS and growing their skills in this area [Knowledge and Skills], (2) connecting with other participants and receiving support [Connection], and (3) reducing feelings of burnout and potential turnover [Well-Being]. Regarding the first theme of Knowledge and Skills, participants in the study remarked on the professional growth they experienced through participation: “I could feel myself changing. You know when I’m meeting with folks, whether it was the teacher for our pre-K program or another person in our company, I could feel that change a little bit. Not every day, not every time, sometimes I was just like, you know, full throttle ahead. But I’m finding more often than not, I have a little bit more of a reflective nature.” [B7]. Participants often described how the RS series strengthened their capacity to use specific reflective skills in supervisory interactions. For example, a participant noted changes in their approach to supervision, while illustrating that becoming a reflective supervisor is a lifelong process: “I think there was positive outcomes across the board. I mean it, really, this practice really got me to pause a lot and sit in silence a little bit more than maybe I have in the past, where I just want to talk, figure out the answer. And really just again let [teachers] follow their own journey and guide them versus tell them. And I think that has translated professionally and personally, though I’m still on that journey, working towards doing it with fidelity.” [B6]. Another participant highlighted how they enjoyed learning a specific reflective strategy used for difficult conversations: “The spotlighting, that piece of reflective supervision was like ‘that’s the tool I’ve been missing.’ How do you, how can you be reflective, empathetic, and understanding but still say, but I need you to do this, you know, without it being like… ‘I’m in charge, you do what I say because I tell you to.’” [B2]. One participant adeptly summed up this theme: “[The RS series] definitely helped me to understand my job more as far as helping the other teachers.” [B8].

The second theme that emerged, Connection, was about how much participants appreciated connecting with others. It is notable that many participants were the only individual in their role within their program, and so the RS series provided an opportunity for outside connections and mutual support. As one participant said, “I was able to interact with other education coordinators that are pretty much in the same boat that I am, like I didn’t feel alone.” [B11]. Supervisors also credited the opportunity to connect with others who shared the same professional role as supporting their growth and well-being: “It was very therapeutic for me. I mean, we all know that the workforce is an issue. We’re all struggling, but just to have that closeness, that community, like we put ourselves out there and we felt very vulnerable, but everyone was super supportive. And sometimes we just sat in silence, and it felt good. Like we didn’t have to have an answer, we didn’t have to [do] that whole problem-solving piece. We wanted to be able to solve the problem to make it better. And sometimes it wasn’t possible, but it allowed me to understand that, and that that’s an okay thing.” [B4]. Relatedly, participation in the RS series reduced feelings of burnout and potential turnover for several participants. The theme of reducing burnout and turnover, Well-Being, appeared to be associated with the increased support derived from participation in the Workshop sessions; this is exemplified by one quote in particular: “People are staying in their positions because they have this time and space. In my small group in particular, there were quite a few of us that were like, ‘I think I’m done’. And that has changed, you know, keeping people in the field. That’s pretty powerful.” [B3]. Notably, the ability to connect with others was less critical for one participant, who held a higher-level leadership role and was able to access a support system within their organization: “We meet often as a group to discuss what’s going on at the center or how can we help that director, teacher... So, I don’t know that it is always reflective, sometimes it is, often it is. So, I think the [Workshop] part wasn’t as helpful for me. I could appreciate having it. But I feel like I already have, like my small group present, so to speak, and that’s something that we do.” [B7].

In addition to the three themes that emerged around acceptability, the majority of participants also expressed that they would like to continue participating in the Workshop sessions and hoped that there would be additional opportunities in the future, as well as opportunities for other education coordinators to participate, especially those who are new to the role. We considered this another favorable indicator of acceptability. As one participant said clearly, “I just wish there was more of it. I wish it didn’t end. When you have the opportunity to be reflective, I think it just improves everything.” [B5].

Effectiveness

Samples A and B Combined

Quantitative Results.

Prior to model testing, participant age, education, years of experience as a supervisor, years of experience in the ECE field, and dosage (number of Workshop sessions attended) were tested as potential covariates, or predictors of change. When examining if any of the potential covariates were related to the RSRS-A or SRS-A, only one association was statistically significant: education was correlated with the SRS-A at the first Workshop session, r = .53, p = .02 (see Table 3 for bivariate correlations). As such, education was initially included as a covariate in the SRS-A model. Education was not a significant predictor of change in the SRS-A over time, F(2) = .67, p = .50, ηp² = .01, and was thus not considered further.

Significant within-subjects effects were observed for the RSRS-A, F(2) = 11.80, p < .001, ηp² = .46, indicating that participant reports on this assessment changed over the course of six months. Post hoc analysis with a Bonferroni adjustment revealed that supervisors’ perceived use of reflective supervisory competencies increased from Foundations of RS to the last Workshop session, p = .044, and from the first Workshop session to the last Workshop session, p = .001, but that there was no difference from Foundations of RS to the first Workshop session, p = .21, as depicted in Figure 3.

Figure 3. Change in Reflective Supervision Rating Scale-Adapted Scores Over Time.

Note. N = 15. ***p < .001.

Significant within-subjects effects were also observed for the SRS-A, F(2) = 7.65, p = .002, ηp² = .30, indicating that participant reports on this assessment changed over the course of six months. Post hoc analysis with a Bonferroni adjustment revealed that supervisors’ perceived use of reflective supervisory competencies increased from the first Workshop session to the last Workshop session, p = .005, but that there was no difference from Foundations of RS to the first Workshop session, p = .67, or from Foundations of RS to the last Workshop session, p = .07, as depicted in Figure 4.

Figure 4. Change in Supervisor Rating Scale-Adapted Scores Over Time.

Note. N = 19. **p < .001.

Sample B (Pilot Open Trial)

Qualitative Results.

Effectiveness was demonstrated qualitatively in Sample B. Participants explicitly stated that they felt the RS series was effective, in that learning to use RS with staff was beneficial. For example, a participant remarked, “So, where we were, as to where we are now, it’s quite a difference. I could see that it worked. It’s a different, a definite different kind of relationship I have with that staff member now than we did when we started out.” [B1]. Another participant said, “[My approach to supervision] definitely changed and shifted more towards that kind of relationship building, asking, wondering, getting more information. So then it’ll help [teachers] to funnel down to their own conclusion. A hundred percent focusing on relationships and the other side of where somebody could be coming from.” [B3].

Participants were asked to explicitly describe what RS meant to them. The majority of participants’ responses demonstrated deep understanding of the reflective stance. For example: “I would say that [RS is], I think, it’s more teacher-led as far as topic wise. I think it is a tool to establish relationships with the people that you supervise. It is providing a safe place for them to talk about real feelings, whether it’s good feelings or bad feelings. Trying to guide them through their own thought process of how to solve problems and kind of giving back to them. You know a feeling of ‘you know you’re safe here, you can talk to me about what you need to. I know this is hard.’ And celebrate with them also when they, you know, when they complete that journey.” [B1]. Furthermore, all participants discussed specific supervisory competencies during their interviews. For example: “So I’m normally very by the book, you know, everything’s planned out, and I like my order to things. So I tried that out with just letting the other person, the supervisee set the agenda and kind of take it, and it was, it was awkward the first time, but it really has opened up the back and forth between me and the teachers. It’s not so much like, okay from me to the supervisee, ‘This is what you need to know. This is what we need to discuss.’ It’s more of a back-and-forth conversation.” [B9]. One participant remarked: “For example, I have a new [supervisee], and in our supervision the other day, like I know that she has a lot of challenges in her room. But in order to be able to really dive any deeper I needed to broaden the story on her background, and on where she comes from, and those kinds of things.” [B5]. Another participant shared: “This way I’m providing a space that just makes them feel like they’re important to me. I don’t want to meet with them in their class where the children are napping and we’re whispering, and you’ve got the background music. So that to me it is important, to just know that they’re important to me.” [B4].

Data Integration

See Table 6 for our joint display of results, integrating observational, quantitative, and qualitative data sources (described above) across both samples, with exemplar quotes. Data integration demonstrated that professional development focused on RS was largely feasible for ECE supervisors to complete amidst their other job responsibilities, that supervisors liked and wanted training in RS, and that training in RS worked.

Table 6.

Joint Display of Integrated Feasibility, Acceptability, and Effectiveness Findings

Feasibility
Sample A	Sample B	Sample B
Observational Data	Observational Data	Interview Data
Met target attendance, but not target attrition, for Foundations of RS	Met target attendance and attrition for Foundations of RS	“I put [the Workshop sessions] in my calendar, and I stuck to it. I had to miss one because my child was sick. But like, I even said to my husband, because in my role I get called a lot to cover [classes]. And I was like, ‘Well, I have [a session]. So if that happens, I’m gonna have to just take time off because I’m not missing it.’ It was a commitment I wanted. It was really easy to stick to for sure.” [OT5]
Met target attendance and attrition for the Workshop	Met target attendance and attrition for the Workshop
Acceptability
Sample A	Sample A	Sample B
Survey Data	Written Data	Interview Data
High satisfaction scores for Foundations of RS and the Workshop	“Overall, I feel like I learned a lot from this [series] and I greatly enjoyed it.” [PE10]	“I really liked [Foundations of RS] because we were all together, it was very interactive… The [Workshop sessions] I really liked, because it was very intimate. It was the same small group throughout the whole thing. It was a nice little networking piece to see how things are working in other schools, and also knowing that I’m not alone here, so it was just really nice. And I looked forward to coming to those [sessions]. …I would like more [sessions]. Or even like another training to be like, okay, this was level one. Okay, what are we doing for level two? What are we going to do to build on what we’ve already learned?” [OT10]
Effectiveness
Samples A & B	Samples A & B	Sample B
Survey Data	Survey Data	Interview Data
Supervisors’ perceived use of reflective supervisory competencies on the RSRS-A increased from the first Workshop session to the last Workshop session	Supervisors’ perceived use of reflective supervisory competencies on the SRS-A increased from the first Workshop session to the last Workshop session	“So I think in my [supervision] meetings I’m becoming a little bit better at listening. That was kind of like my overarching goal throughout the whole [series]. Every week I’d be like ‘I’m supposed to listen better.’ Because I get so excited that I want to share. So really working on that, and being mindful of ‘am I giving the person the space they need to process what they’re going through’ without sticking a band-aid on it, or sharing in their misery... So that’s one area I think that I have shifted a little bit.” [OT2]

Note. RSRS-A = Reflective Supervision Rating Scale-Adapted (RSRS-A), SRS-A = Supervisor Rating Scale-Adapted.

Discussion

This paper assessed the feasibility, acceptability, and preliminary effectiveness of a professional development series in RS in two ECE samples in Rhode Island, USA: supervisors overseeing multiple types of classrooms, in the context of a community-based program evaluation, and supervisors overseeing state pre-K classrooms, in the context of a pilot open trial. Using data from multiple sources, in this case, two samples, is an important method of ensuring rigor in mixed methods research. Moreover, these two samples were representative of a common situation within our state—that many supervisors work in centers with a range of classroom types—and allowed us to address the empirical question of whether the RS series was appropriate for supervisors supporting different types of classrooms. Notably, the vast majority of supervisors in both samples were white, female, had a bachelor’s or master’s degree, and had been working in ECE for ten or more years. While these demographics are representative of the supervisors in our state, results should be considered within that context.

Although RS has long been considered a best practice in many early childhood settings, the evidence base for this approach has really begun to grow over the past decade, and generally demonstrates that receiving RS has numerous system-wide benefits (Huffhines et al., 2023). The current paper adds to the existing evidence in two important ways. First, the focus on training supervisors in RS addresses the previously neglected yet essential first step of building an effective supervisory workforce, who can then go on to provide the high-quality RS that professionals need and deserve and most importantly, have been shown to benefit from. Second, exploring whether such training is feasible, acceptable, and effective in ECE specifically has important implications for a system that millions of young children and families interact with every day, and that local, state, and federal governments heavily invest in (Alawsaj et al., 2023; Slovick et al., 2023). Overall, findings indicated that the RS series was feasible to implement, acceptable to participants, and effective in developing supervisory competencies. This was true regardless of ECE context, with some nuances, suggesting that the RS series may be appropriate for supervisors in a variety of ECE settings.

Feasibility

The RS series generally demonstrated feasibility. Attendance rates for both components of the series (Foundations of RS and the Workshop) in both samples met our a priori threshold of 70% or higher, suggesting that supervisors wanted to/were able to attend, regardless of ECE setting. Furthermore, in Sample B, the open trial, there was no attrition from Foundations of RS or the Workshop, meeting our a priori threshold of 20% or lower. However, in Sample A, the program evaluation, attrition from Foundations of RS exceeded the 20% threshold. Attrition from the Workshop, in contrast, remained less than 20%. It is important to understand how differences between the two samples may account for greater attrition from Foundations of RS in one sample but not the other. One possible explanation is that Sample B completed Foundations of RS in a single day, while Sample A completed Foundations of RS over two to four days. While all offerings were eight hours and identical in content, it may have been easier for supervisors to commit to an entire day of professional development than to return on multiple occasions. Offering Foundations of RS over multiple days also increases the chances that other obligations will arise for supervisors, such as the need to provide classroom coverage. Another possible explanation is whether time for professional development is protected across different settings. For instance, supervisors who serve primarily private for-profit or non-profit classrooms, like Sample A, tend to have less built-in or protected time for professional development than supervisors who serve primarily state pre-K classrooms, like Sample B. Understanding the barriers to completing Foundations of RS, particularly for supervisors who do not work in state pre-K, will be key to making this professional development opportunity more accessible within ECE.

Completing Foundations of RS was required to participate in the subsequent Workshop for both samples. Sample A participants first enrolled in Foundations of RS. Those who completed Foundations of RS were then invited to participate in the Workshop at a later date. For Sample A, the two curricula comprising the RS series were conceptualized as separate—though related—professional development offerings, with the idea being that more supervisors would enroll in Foundations of RS and a smaller group of supervisors who completed Foundations of RS would enroll in the Workshop. Indeed, there were fewer Workshop slots offered than slots in Foundations of RS. Sample B participants, on the other hand, were enrolled in a pilot open trial with the understanding that they would participate in both components, given that this was a research study with the goal of evaluating the RS series as a whole. Given how we conceptualized the flow from Foundations of RS to the Workshop in each sample, we did not assess feasibility by the percentage of Foundations of RS completers that subsequently enrolled in the Workshop. Differences in how we conceptualized the RS series for each sample likely contributed to a greater percentage of participants from Sample B (92%) enrolling in the Workshop compared to Sample A (33%). It may be useful to explicitly evaluate this metric of feasibility in future studies, to understand the factors that contribute to enrollment and completion of the Workshop following Foundations of RS, and determine if there are differences based on ECE setting or other demographic variables. Notably, it is possible that less attrition was observed for the Workshop in both samples because only the most invested supervisors who wanted to learn more about RS enrolled in that component of the series. However, as individual session attendance rates and qualitative interviews with Sample B demonstrated, a supervisor may be motivated to attend, yet still must contend with other responsibilities and obligations within their programs and personal lives.

Acceptability

The RS series demonstrated acceptability, in that participants in both samples were highly satisfied with both components of the series (Foundations of RS and the Workshop). Responses on a rating scale, written responses, and statements made in individual interviews each confirmed satisfaction. Though themes varied slightly between Sample A and Sample B, participants universally found that the RS series helped them to learn key supervisory skills and strategies, use those skills and strategies in supervisory interactions in their programs, and foster connections with other participants who shared their professional role. Participants’ responses confirmed that the attention paid in Foundations of RS to balancing content with experiential practices was important for learning. This finding is consistent with an evidence-based approach to professional development based on multiple research syntheses and meta-analyses of adult learning, which emphasizes the importance of active learner involvement (Dunst & Trivette, 2009). Furthermore, the warm, respectful learning environment created by facilitators helped all participants have a positive experience; encouragingly, this was remarked upon regardless of the delivery modality. In some instances, supervisors witnessed a decrease in their own feelings of burnout and desire to leave their jobs due to participation in the RS series. Notably, there was little variability in demographic factors across the two samples, and thus we could not determine whether such factors may influence participants’ satisfaction with the RS series or their sense of stress and burnout following participation; this is an important area of exploration for future research.

Effectiveness

The RS series demonstrated preliminary effectiveness, per change in reflective supervisory competencies over time within the combined sample, and per responses in individual interviews in Sample B, the open trial. On two measures, self-reported supervisory competencies were shown to increase from the first Workshop session to the last Workshop session, indicating that within six months, participants perceived growth in their supervisory competencies; moreover, these effects were large. This finding is consistent with the other known study testing this professional development series in RS, but in a sample of home visiting supervisors (Low et al., 2018). Notably, on both measures, participants rated their supervisory competencies somewhat higher at the baseline assessment than they did at the first Workshop session. This pattern may reflect the Dunning-Kruger effect, which can be summarized as “people don’t know what they don’t know” (Dunning, 2011). In this context, supervisors may initially rate their supervisory competencies highly until they learn, through the foundational training, what it looks like to hold a reflective stance and enact reflective skills and strategies in supervisory interactions. Their ratings at the first Workshop session may therefore be a more accurate depiction of their competencies.

Evidence of the RS series’ effectiveness is further bolstered by qualitative findings. Not only did participants in Sample B report that they believed the series to be useful in improving their supervisory practices, when asked to discuss what RS meant to them, and what they thought the core components of RS were, the majority answered in a detailed way that conveyed genuine understanding. Furthermore, supervisors also described, unprompted, the specific skills that they now use in supervisory interactions that they didn’t before, such as attunement, negative capability, and collaborative problem solving. These findings align with those of Shea and colleagues (2022), in their evaluation of a statewide infant and early childcare education professional development system. Participants in that study included six supervisors and program managers, along with grant specialists and consultants, who participated in monthly reflective learning groups for 12 months; reflective practice self-efficacy increased over the course of the year, and participants also reported improvements in their reflective practice skills via written responses on a survey (Shea et al., 2022). Although the focus of Shea and colleagues’ study was not on training supervising professionals to provide RS to others, it did include participants in similar roles in a similar setting, and collectively, our work demonstrates that reflective practices can be successfully taught through professional development for ECE leaders. However, it remains unclear whether training in RS may affect reflective supervisory competencies for all ECE supervisors, given the homogenous nature of our samples. Future research should examine effectiveness of the RS series for supervisors with diverse identities, backgrounds, experiences, and geographic locations.

Strengths, Limitations, and Future Directions

Although this paper included data from two contexts, a program evaluation and a pilot open trial, and represents a novel effort in the field to move beyond examining the effects of receiving RS to evaluating the effects of training in RS, several limitations should be noted. First, the program evaluation was not a research study, and therefore its methods are not as comprehensive as those of the pilot open trial. However, it is important that taken together, the program evaluation and pilot open trial corroborate each other’s findings, while also highlighting some differences between ECE settings (e.g., state pre-K versus other programs) that have useful implications for implementation. Future studies should adopt even more rigorous methodological approaches than are presented here. RCTs for example, will allow the field to compare outcomes within programs of supervisors who complete the RS series with those who do not complete the series (or are waiting to complete the series). Second, while we selected two commonly used measures for our effectiveness aim, measurement of reflective supervisory competencies is still in its infancy, and remains a moving target for the field; indeed, many have written about the difficulties of measuring reflective practices, and the need for more sophisticated methods (Tomlin & Heller, 2016; Watson et al., 2016). Hence, our inclusion of both quantitative and qualitative methods in this paper. Effectiveness findings based on observational methods will be presented in a forthcoming paper. Third, and relatedly, to fully assess the effectiveness of training supervisors in RS, change in supervisee, family, and child outcomes must also be measured, which was beyond the scope of these efforts, but will be addressed in our RCT, which is currently ongoing at the time of writing. Fourth, we were unable to examine differences on outcomes between supervisors who completed the RS series virtually versus in-person, given that delivery modality was conflated with occurrence of the pandemic and sample; future studies could test the potential impact of modality.

Several limitations related to issues of diversity must also be mentioned. Future research should focus on system-wide changes that may be brought about by RS, such as increased discussion of social identities, power, privilege, and bias in supervisory interactions, which may in turn enhance teachers’ sense of belonging and promote greater representation of diverse identities in the workforce as well as build up inclusive practices with children, ultimately advancing equity within early learning centers (Noroña et al., 2012). While the RS series we investigated did attend to these issues through its curricula, we imagine that additional, targeted training is likely necessarily for observable change in this domain, especially for supervisors who have not participated in professional development focused on these issues. Additionally, although our samples were diverse in terms of supervisor age and ECE workforce experience, which we view as a strength for the make-up of the Workshop groups, both of our samples lacked racial and ethnic diversity. While this was a reflection of who comprises the ECE supervisory workforce in our state, we believe that it will be important to understand how professionals of color might view and benefit from the RS series. Moreover, the demographic data we collected for both samples was limited in terms of capturing the wide range of identities that may impact supervisors’ sense of belonging within ECE settings or their experiences of oppression, stress, and burnout in their workplaces. Understanding whether and how training in RS may affect these experiences, as well as the experiences of the teachers who are being supervised—all of whom have their own complex and intersecting identities and histories—is an important future research direction (Eaves et al., 2022).

Conclusion

RS, no matter the evidence of its impact, will not reach the early childhood-serving professionals who need it until we understand how to effectively train their supervisors. Through a community-based program evaluation and a pilot open trial, we established that a particular professional development series in RS, developed from best practice guidelines and designed for optimal adult learning, was not only feasible and acceptable for supervising professionals in different types of ECE settings, but showed signs of effectiveness. While more research is needed to fully test the RS series, this paper represents a critical first step in the implementation of RS within ECE.