What’s the headline on your mind right now? How reflection guides simulation-based faculty development in a master class

Abstract

Introduction

The demand for highly skilled simulation-based healthcare educators (SBEs) is growing. SBEs charged with developing other SBEs need to be able to model and conduct high-quality feedback conversations and ‘debrief the debriefing’. Direct, non-threatening feedback is one of the strongest predictors of improved performance in health professions education. However, it is a difficult skill to develop. Developing SBEs who can coach and support other SBEs is an important part of the faculty development pipeline. Yet we know little about how they get better at skilled feedback and the ability to reflect on it. There is scant evidence about their thoughts, feelings and dilemmas about this advanced learning process. To address this gap, we examined advanced SBE’s subjective experience as they grappled with challenges in a 4-day advanced SBE course. Their reflections will help target faculty development efforts.

Methods

Using a repeated, identical free-writing task, we asked “What is the headline for what is on your mind right now?”

Results

A five-theme mosaic of self-guiding reflections emerged: (1) metacognitions about one’s learning process, (2) evaluations of sessions or tools, (3) notes to self, (4) anticipations of applying the new skills in the future, and (5) tolerating the tension between pleasant and unpleasant emotions.

Conclusions

The results extend simulation-based education science by advocating the motivational role of noting inconsistencies between one’s intention and impact and the central role of self-regulation, emotion, and experiencing feedback and debriefing from multiple perspectives for improving advanced skills of SBEs. Recommendations for faculty development are discussed.

Introduction

The demand for highly skilled simulation-based healthcare educators (SBEs) is growing as simulation is integrated into curricula and hospital training at all levels.^{1 2} Regulatory bodies in healthcare increasingly require or accept simulation as part of accreditation, licensure or maintenance of certification.³ Standards for SBE faculty development and consistency are leading institutions to seek ways to develop SBEs and SBEs of SBEs.^4–6 As a consequence, SBE faculty development methodology is attracting more attention, particularly regarding the advancement of debriefing skills.^7–9

SBEs charged with developing other SBEs need to be able to model and conduct high-quality feedback conversations and ‘debrief the debriefing’ with colleagues and peers seeking to improve their debriefing skills. Direct, non-threatening feedback is one of the strongest predictors of improved performance in health professions education.^10–15 However, developing the skill to initiate and sustain debriefings and feedback conversations with peers that includes honest but non-threatening feedback is difficult and eludes even some advanced SBEs.¹⁶

Although developing SBEs capable of developing other SBEs is an important part of the faculty development pipeline,^{8 17} not much is known how SBEs move towards highly skilled feedback practice and the ability to reflect on it, particularly their subjective experience of the learning process. While the skills required to teach and learn using reflective practice are well described,13–15 18–36 we know little about SBEs’ use of self-regulation—a combination of metacognition, strategic planning and motivation^37–39—to develop and sustain their own feedback skills.^{38 40 41} This lack of insight into what it is like to struggle with this advanced learning process—the subjective experience—impedes targeting and facilitating effective faculty development programmes.

To address the gaps in our current understanding of how SBEs experience the learning process of developing as skilled debriefers and ‘debriefers-of-debriefers’, this study sought to capture what captivated or concerned experienced SBEs as they moved through a 4-day intensive advanced simulation instructor skills development course. Using 2 min free writing tasks for each learner called ‘headlines’, we systematically ‘biopsied’ learners’ subjective experience throughout the course. The primary goal of this qualitative study was to examine how advanced SBEs experience the core challenges of developing honest but non-threatening feedback conversation skills focused on transparency, curiosity and respect. Unlike conventional programme evaluations, asking educators in development to reflect on their thoughts and feelings about the ongoing learning process can reveal hidden dilemmas, insights and self-regulation processes.^{42 43} How do advanced SBEs reflect on the efficacy of their feedback skills within debriefing? What do they struggle with while aiming to improve these skills? And what self-regulation strategies do they use to sustain newly acquired skills? Answers to these questions will help develop and target faculty development efforts for clinical faculty. Furthermore, learning to master empathic yet rigorous feedback conversations, ‘caring personally while challenging directly’⁴⁴ has important secondary benefits for healthcare learning. Instructors and learners developing the skill to have ‘difficult conversations’⁴⁵ in debriefings is itself a simulation for learning conversations in the real workplace. ‘Difficult conversations’ in debriefings explore performance and facts of the simulation, and  also feelings, mental models and identity threats in both single discipline and interprofessional contexts.¹⁴ Difficult feedback conversations within debriefings prepare both instructors and learners for such conversations outside of the simulation context.⁴⁶

Methods

The Advanced Instructor Course

The Advanced Instructor Course (AIC) is an immersive peer-to-peer and mentor-guided incubator for high-level simulation-based education skills, especially debriefing and feedback on debriefing. Educators in the course support each other in developing these skills primarily through three mechanisms (see table 1 for an overview and online supplementary table 1 for a detailed course description):

Table 1.

Overview of the Advanced simulation Instructor Course

Session	Description
Simulation scenario	Design, prebrief, conduct and debrief a simulation scenario with feedback on the scenario and debriefing Experiential perspectives via multiple iterations: as an instructor, participant and observer giving feedback
Learning pathway grid	A structured group analysis of a previous debriefing to strengthen faculty self-reframing skills, identify what worked and what did not during a debriefing, and think through and rehearse alternatives to enhance future debriefings87 88 Explores mismatch between good intentions of the debriefer and unwelcome impact on the learners45 Based on a prepared two-column case study of previous debriefing,59 89 that is, a representation of what debriefer said and did in one column and what she/he thought and felt (but did not say) in a parallel, second column Originally developed by Action Design (www.actiondesign.com) and adapted by lead JWR for debriefing context Experiential perspectives via multiple iterations: as a case writer, group member and peer facilitator of the discussion
Simulated debriefing	Debriefing a simulated case14 90; the case is provided by a recording of a clinical scenario. Group members then assume roles of people in the recording and one person debriefs them The exercise allows participants to practise debriefing without the time and resource demands of simulation Facilitators and peers ‘debrief the debriefing’ Experiential perspectives via multiple iterations: as an instructor, participant and observer giving feedback
Lecture-based inputs	Provide in-depth introduction or refresher of topics for advanced simulation instructors (eg, on peer feedback, difficult debriefing situations, adjusting facets of realism in simulation)
Expert consulting (elective)	One-hour informal, in-depth consulting rounds with faculty and peers on topics such as objective-oriented debriefing, research and assessment, faculty development, strategy and negotiation
Realism and improvisation workshop Non-clinical teamwork	Illustrate the meaning of realism and improvisation during simulation Practise key improv skills such “The offer” and “Yes and…” Explore how to debrief teamwork and team learning in a non-clinical challenge

For a more detailed course description, see online supplementary table 1.

1. a curriculum that invites repeated confrontation with one’s own inconsistencies as an educator^47–49 to mobilise change in practice,^50–52 in a context that aims to be psychologically safe⁵³ yet challenging⁵²;

2. experiencing debriefing and feedback from multiple perspectives: simulation designer and director, debriefer, participant, observer providing feedback^54–56;

3. deliberate practice⁵⁷ of simulation design, debriefing and feedback on debriefing, all with peer and mentor feedback.

Using immersive, experience-based learning, the AIC consists of repeated exercises with feedback requiring both reflection in action⁴⁸ (examining and reframing one’s own cognitions in the moment) and reflection on action⁴⁸ (examining and reframing ones’ own previous cognitions), as well as lectures and informal discussions on simulation-related topics. A central goal of the course is to strengthen SBEs’ ability to reflect on their taken-for-granted cognitive routines, assumptions, emotional reactions and their behavioural consequences. The course aspires to allow SBEs to identify the mismatches between the intent of their actions and the impact. Being able to detect and correct intent-impact mismatches either in the past or in the present, it is hoped, strengthens their ability to identify and improve patterns in how they interact with learners, colleagues and patients.^{45 58} The AIC included 18 scheduled sessions, some of which were attended by every participant whereas others were electives.

The instructional design seeks to leverage the power of confronting one’s own inconsistencies, a method recognised in various theories of experiential learning^{50 58 59} as a way to transform one’s perspectives. As they practise feedback conversations, many educators in the course encounter three central contradictions within themselves, documented in the social psychology and action learning literatures.^58–61 First, they tend to espouse that honest, direct critique is good, but in practice, frequently camouflage critique behind leading or guess-what-I-am thinking questions (eg, “Wouldn’t it have been better to…?”). Second, participants often find that their good intentions (eg, to help learners do better in the future) during debriefings can backfire in baffling and unforeseen ways (eg, learners become defensive or angry), illuminating a mismatch between their well-meaning goals and the effect.^{45 61} Third, they espouse that curiosity and positive regard for learners is an effective teaching strategy,⁵⁹ but in practice they judge the person (not just their action) negatively, and lose their curiosity about why learners do not meet an expected standard.⁶² Learning to identify and ‘embrace’ these mismatches or ‘hypocrisy’ is often a springboard to launch a new level of practice.⁴⁷

At the time of data collection, the course faculty was researching, practising and publishing on learner experiences addressed in the course, allowing them to bring empirical, practical and theoretical insights to supporting learners. Areas of overlap between course design and their expertise included debriefing,11 13 22 63 reflective practice (including their own practice)^{16 60} and realism in simulation.^64–66 In addition, they brought practical experience of collectively having conducted more than 6000 debriefings, and through their SBE training activities, had observed and provided feedback on over 2500 debriefings by instructors with a broad range of debriefing styles and skill levels from Asia, Oceania, North America, Europe, Central and South America.

Study design

This study used an exploratory, mixed method approach. Using experience sampling,⁶⁷ we tracked what captivated or concerned experienced clinical faculty as they moved through the AIC: we designed brief, written reflections after each of module of the course to systematically stimulate thinking about learning. These repeated, identical prompts were intended to generate insights about experience over time and were analysed for this study. MK took the lead in designing the study and the data collection tool; she was neither involved as a designer nor instructor of the AIC. JWR provided guidance in study design and data analysis. She was one of the designers and instructors of the AIC.

Participants

Participants were 25 clinical faculty members from hospitals, nursing and medical schools around the world. They had previously attended an educator workshop in healthcare simulation. After gaining experience as SBEs they could enrol in the AIC to refresh and extend their simulation-based education skills and repertoire. The participants came from eight different countries.

At the beginning of the course, all participants were invited to take part in the study by granting us access to their written reflections. They were explicitly informed that if they decided not to participate, they could inform the course co-ordinator who did not belong to the teaching faculty and who would later inform MK about which study identification (ID) number’s written reflections not to include into the study. All course participants decided to participate.

Data collection

For the purpose of this study, we developed a straightforward 2 min free writing task which we called ‘headline’: after each exercise throughout the course, participants received a sheet of paper entitled ‘headline’ which included the following open-ended question: “What is the headline for what is on your mind right now?” and the prompt ‘headline’, followed by a blank line indicating participants should answer the question with a few words only (see online supplementary figure 1). Based on systemic-constructivist reflection techniques,⁶⁸ this task was intended to stimulate reflection, ‘construction’ of new understandings, as well as to allow for verbalising the reflection in a succinct way. Being very short and crisp, the headline prompt permitted for multiple, longitudinal measures of subjective experiences throughout the course. Data collection was anonymous and confidential. Learners choose a unique ID number. This allowed for tracking participants’ headlines throughout the course. Assignment of participants’ names to their IDs or headlines was not possible. The ‘headline’ reflection is not typically part of the AIC and was introduced solely for the purpose of this study.

Data analysis

We applied a multistep, thematic analysis^{69 70} to identify topics that were evident during the reflections. Each headline was considered one analytic unit. Following procedures for linking inductive and theory-driven coding described by Ibarra⁷¹ and others,^{72 73} we started inductively by reviewing and paraphrasing headline after headline and generating a list of rough categories in an open-coding process. We then reviewed rough categories and identified clusters of categories, which we discussed and revised. This resulted in a preliminary coding list with categories describing the form in which the content of the headline was presented as well as codes describing the themes represented in the headlines. We then worked more deductively in an iterative process of moving back and forth between the original headlines, their assumptions, the relevant literature and the emerging categories. Finally, we analysed whether the categories were related to certain components of the course. This process is described in detail in table 2.

Table 2.

Data analysis procedure

Step	Thematic analysis procedure
1	MK typewrote all headlines.
2	MK reviewed and paraphrased headline after headline and generated a list of rough categories in an open-coding process.
3	MK reviewed rough categories and identified clusters of categories, which were discussed and revised with the second author and resulted in a preliminary coding list with categories describing the form in which the content of the headline was presented as well as codes describing the themes represented in the headlines.
4	JWR and MK used an iterative process of moving back and forth between the original headlines, their assumptions, sensitising concepts of the relevant literature (eg, self-regulation) and the emerging categories. Following the process described by Ibarra71 and others,72 they compared the headline data, their emerging categories and the literature mainly on self-regulation, learning, reflection, training and emotion37 43 91–96 to guide decisions about the final categories that would describe the data best. For example, they noted that a considerable number of headlines reflected feelings. After consulting the literature on emotion, they classified most these feelings according to the circumplex model of affect95 into pleasant versus unpleasant and activated versus deactivated emotions. A similar procedure was applied for the other emerging categories, in particular for those representing well-acknowledged learning conditions such as psychological safety and deliberate practice.57 96 By generating and updating a codebook, step 4 resulted in a final list of refined and confirmed codes describing the headlines best (table 2). Following a procedure described by Miles and Huberman (pp. 55–66),70 JWR and MK clumped descriptive codes into categories describing the form in which the headline was presented.
5	MK applied the list of final categories for re-coding the complete data set. Multiple coding, that is, assigning more than one process and content code to one single headline, was possible.
6	MK sorted all coded headlines with respect to the exercise they referred to and the day and time the exercise took place.
7	As a check on JWR’s and MK’S developing understanding, they confirmed that an independent coder, an emergency medicine physician and advanced simulation instructor who had taken the  Advanced Instructor Course and was a simulation fellow could identify the categories in the data.
8	Determining absolute frequencies for all categories.
9	Analysis of relationship between final categories and respective components of the course. For this analysis, those 18 headlines (4.35%) lacking a reference to the respective session were excluded, leaving 396 headlines. We realised that the headlines’ baseline and the response rate were uneven. For example, the learners had the possibility to attend nine highly interactive, small group session (scenario, learning pathway grids, simulated debriefing), eight lecture-based inputs and other plenum interactions, and seven expert coaching sessions during their free periods. While participation during the interactive, small group sessions was mandatory, attendance of the lecture-based inputs was not strictly monitored, and participation in the expert coaching sessions was optional. Thus, we had no information whether learners actually participated in all possible sessions. We had also no information whether learners wrote a ‘headline’ after every session they participated in. Due to this lack of reliable information on how many times a learner participated in a certain session and how many times of which she/he completed a ‘headline’, we decided to focus on the absolute numbers of themes per session type as they occurred rather than relating them to potentially biased base rate.

Results

The mean number of headlines provided was 16.44 for each participant (SD=1.66). A total of 414 headlines were obtained. Out of the 414 headlines, 18 (4.35%) did not include a reference to the exercise on which it was a reflection. Four headlines (0.97%) referred to more than one exercise.

Five themes emerged from learners’ reflections on trying to improve their debriefing and feedback skills (figure 1).

Figure 1.

A five-facet mosaic of learners’ reflections while improving their debriefing skills. Each facet represents a theme induced from coding and thematic analysis of the headlines. Below each theme is an example ‘headline’.

Metacognitions of one’s learning process

The first theme emerged as specific metacognitions about one’s individual learning process, such as monitoring one’s learning progress and identifying current performance gaps (eg, “I’m curious to know why I’m not more curious”), developing ideas about how to close these gaps (eg, “Practice really helps me. Listening to others practice is surprisingly useful…”) or reflecting on one’s educator identity (eg, “I cannot/need not know everything”). Online supplementary table 2 provides more examples. Metacognitions were broadly distributed over all types of session (figure 2).

Figure 2.

Results of analysing headline categories (absolute frequencies) as per Advanced Instructor Course session type (scenario, LPG, simulated debriefing, expert coaching and lecture-based inputs). LPG, learning pathway grid.

Evaluations of tools, sessions and performances

The second theme included evaluations of the usefulness, value or quality of a session, tool or skills demonstrated by the AIC faculty. While headlines of this facet may represent the typical content of formal course evaluations, such as critically reviewing a session, others went beyond simple evaluation to include explicit acknowledgements of the challenges, complexities and difficulties that were involved in learning (eg, “opening up to critique is brave”). A number of headlines indicated that the SBEs reflected deeply on how to use certain tools and on what it was in particular that other participants or the teaching faculty did during the course that was perceived as helpful. Evaluations were the leading theme group after lecture-based inputs (61 times, eg, “This is good stuff… I want more”) and after learning pathway grids (LPGs) (39 times, eg, “Excellent learning tool”).

Notes to self

The third theme—notes to self—emerged from participants’ reflections on the specific concepts that were taught during the AIC, such as curiosity, honesty, psychological safety and cognitive frames. We called these reflections a ‘note to self’. They included remarks exhibiting an ‘aha!’ moment or new understanding: the importance of a concept, of a similarity between concepts and of the nature of a concept. They also included instructions about how to do something in the future based on specific concepts taught during the AIC. In some cases, these notes to self were structured as a sequence of steps (eg, ‘orient, preview, reflect, reframe’), and in other cases they seemed to be designed as ‘personal contingency models’ suggesting situation-specific actions (eg, “when I don’t know → explore”). Notes to self occurred most frequently after lecture-based inputs (42 times, eg, “Do no harm”) and were—compared with the other four themes—the leading theme group after simulated debriefings (31 times, eg, “Give your opinion”; “Practice is key”) and scenarios (17 times, eg, “Knowing your scenario perfectly helps you debrief it”).

Anticipations of applying the learnt skills in the future

The fourth theme—anticipations of applying the learnt skills in the future—emerged from headlines predicting or foreseeing how skills acquired in the course would be applied in the future. These anticipations included remarks in which participants happily looked forward to applying something they had learnt in the AIC, concerns about how to apply skills in the future and speculation about their motivation to apply the skills. Anticipations occurred most frequently after lecture-based inputs (10 times, eg, “Hope I can put it all together and help those around me to facilitate Sim in all the areas I’m working in”) and after LPGs (5 times, eg, “LPG at home”).

Emotions in the learning process

The fifth theme—emotions in the learning process—related to affect triggered by learning activities. We applied an existing conceptual model to analyse this single theme once we had identified emotions as code. Using the circumplex, two-axis model of emotion,⁷⁴ we coded the emotions in the headlines as pleasant and unpleasant sentiments that were either activated or deactivated. Learners described more pleasant than unpleasant and more energetic than deactivated emotions. One of the emotional states that was frequently mentioned was the feeling of exhaustion. Emotions were noted most frequently after lecture-based inputs (16 times, eg, “I am happy to have a better understanding of reliability and validity”), scenarios (11 times, eg, “Fried brains. But good”) and simulated debriefings (6 times, eg, “Phew! What a relief!”).

Discussion

This study explored what advanced SBEs experienced while trying to enhance their feedback and debriefing skills. The advanced SBEs’ reflections, concerns and emotions provide insight on the rewards and challenges of this work as well as guidance on how to target SBE faculty development.

First, our data suggest that providing challenges structured to activate and sustain the self-regulation can yield concrete self-generated guidelines for self-development. In alignment with theories of self-regulation in learning,^{43 75} we found that advanced SBEs’ metacognitions focused on struggling with and making sense of their own educational dilemmas and challenges. Opportunities for self-monitoring,⁷⁶ self-regulation^77–79 and first person ‘research’⁸⁰ (examining one’s own learning processes systematically) appeared to drive a host of insights such as rules to self and planning for future practice.

SBEs in the course generated explicit guidelines on the ‘microskills’ or subparts of effective feedback and debriefing. These self-generated guidelines allow advanced SBEs to better develop colleagues’ skills in targeted and effective ways. As they become better able to identify and improve these microskills in their own practice, they can then identify and discuss them with peers and colleagues they are developing. This breaks advanced practice down into manageable, repeatable steps as demonstrated by the ‘notes to self’ (eg, for leading debriefings: ‘orient, preview, reflect, reframe’). It allows faculty-in-development to give names or labels to some of the almost magical-seeming tacit knowledge of advanced practice. It helps SBEs transform the invisible guiding rules of expert facilitators or educators into goals and standards for self-improvement and peer improvement. The implication for SBE faculty development programmes? Building in explicit, regular opportunities (as the headlines did) for SBEs to monitor their learning (eg, “I can step out of my comfort zone!”), identify performance gaps (eg, “need to state my point of view more openly”) and how to close them (eg, “practice and watching others practice helps me”) may help initiate this regular reflection.

Second, faculty development programmes can enhance skills by providing rigorous ways for educators to contrast multiple perspectives on the same activity, such as debriefer, participant and observer. For example, emotions generated by experiencing that one’s feelings in the debriefing were not validated were uncomfortable but motivating (eg, “Have just experienced being in the hot seat of the simulated debriefing and not feeling like my feelings were validated and therefore being too distracted to concentrate on how to get better, I realise the overwhelming importance as a debriefer to maintain an engaging context for learning—maintaining/showing the basic assumption, validating participants' feelings, sharing my point of view”). This awareness of the impact from different perspectives, in turn, might have generated the frequent occurrence of ‘notes to self’ (eg, ‘being honest is the best cure for a bad debriefing’) after simulated debriefings and scenarios. Part of the didactic curriculum focused transforming harsh or hidden judgement into fair or ‘good judgement’ as a way to reduce the intent-impact mismatch. Most likely, it was the multiperspectival experiencing of harsh, hidden or good judgement that seemed to help participants move along the path from hidden judgement and camouflaging direct critique to embracing non-threatening fair judgement as part of their repertoire (‘Honesty makes for more productive learning’).

Third, advanced faculty development programmes need to help educators plan and find ways to practise their self-prescriptions for improvement. The findings reveal that SBEs are concerned about being able to apply the learnt skills in the future (eg, “How much of this will I apply?”; “I’m worried though that no one else at my institution knows how to do this and I may not be able to objectively use this tool to evaluate one of my debriefers”). Therefore, faculty development programmes should provide participants with an active role in ‘relapse prevention’-like identifying how to maintain and improve their skills (eg, regular feedback opportunities, triggers for self-monitoring, communities of practice). Wish, Outcome, Obstacles, Plan approaches,⁸¹ online communities for skill practice or local peer-group ‘work-outs’^{8 16} could play a role.

Fourth, this study indicates that normalising and leaving room for emotions as part of the learning process may support educator learning and development. The emotions described in the headlines suggest that the SBE’s development is not merely a cognitive process but involves a broad range of feelings, both pleasant and unpleasant as well as energetic and deactivated. While the critical role of affect for learning and performance has long been acknowledged,^82–84 our study highlights the salience of contradictory emotions for advanced SBE development, such as feeling ‘exhausted and exhilarated’, ‘safe to be uncomfortable’, ‘free and constrained’.^{85 86} This underscores the importance of providing psychological safety for learning which gives learners a protected space to process opposing and unpleasant feelings.⁵³ A ‘safe container’ may also normalise and encourage the ‘struggle’ of learning (eg, “I am not alone with being nervous”).³⁸ More research is required as to whether explicit discussions of emerging feelings during training facilitates learning.³³ These findings also indicate that allowing for more downtime during intense faculty development courses may help learners manage their exhaustion, and integrate new insights into experiments with new approaches.

Further research is also needed to systematically analyse how a mixture of both instructor-designed structure of repeated opportunities to practise reflection and room for self-regulated reflection may interact in facilitating learning. Our findings suggest that shifting perspectives on the same educational activity (eg, debriefing) via alternating first, second and third person practice⁸⁰ may deepen and accelerate learning new debriefing skills and learning to learn new skills. For example, for improving debriefing skills, first person practice involves performing a debriefing, second person practice involves being debriefed and third person practice involves watching faculty and peers debriefing others. More research on the optimal mixture of educational activities with first, second and third person practice would help to titrate the mixture of these activities in faculty development programmes.

This study has limitations. We investigated a small sample of clinical faculty as learners as they moved through one course with a focus on simulation-based education. Results are based on self-reports and may depend on the course quality. Further research might test the sensitivity of the headline methodology for other populations than experienced SBEs, for example medical and nursing students.

In sum, this study surfaced a mosaic of reflections illustrating that advanced SBEs monitor their own learning process, develop helpful notes to self, experience a broad range of emotions, deeply reflect on the use of course sessions and content, and are concerned about their ability to apply the new skills after the course. The results extend simulation-based education science by demonstrating (1) the role of experiencing feedback and debriefing from multiple perspectives (eg, debriefer, learner, observer) in close temporal proximity; (2) the motivational role of noting inconsistencies between one’s intention as a teacher and the impact on the learner; (3) central role of emotion in self-regulation as advanced SBEs attempt to improve their own skills.