Assessing healthcare simulation facilitation using a competency‐based tool derived from practice in low‐resource settings

Summary

Introduction

The worldwide expansion in healthcare simulation training includes accelerated uptake in low‐resource settings. Until recently, no framework has specifically delineated the competencies underpinning effective facilitation practice in low‐resource settings. We describe the development of the Facilitation Behavioural Assessment Tool for simulation facilitation training and report reliability in scoring facilitation performance. This tool was informed by healthcare simulation facilitation practice in low‐resource settings.

Methods

The tool has 32 facilitation competencies, organised across three performance categories (techniques, artistry and values) and a three‐point scale is used for scoring. Following a short, self‐directed online training module, participants scored three videos that depicted facilitation performance at three levels. Videos were presented in a random order. Intraclass correlations and internal consistency with Cronbach's α were calculated. A random intercepts 3 × 3 linear mixed model assessed discrimination across the three levels of facilitation performance and the influence of previous facilitation on scoring.

Results

In total, 104 participants from 29 countries completed rater training and scored at least one video. The inter‐rater reliability was 0.73 (95%CI 0.66–0.79) and 0.89 (95%CI 0.85–0.92) for the intraclass correlation coefficient 2 and intraclass correlation coefficient 2k, respectively. Cronbach's α was 0.84 (95%CI 0.79–0.89) for the positive video; 0.84 (95%CI 0.78–0.88) for the mixed video; and 0.91(95%CI 0.87–0.93) for the negative video. Previous simulation facilitation experience did not affect the ability to distinguish between the videos meaningfully, but novice facilitators scored facilitation behaviours higher for mixed and negative videos compared with participants with intermediate and high levels of experience.

Discussion

Our study shows that suitable reliability and internal consistency can be achieved when using the Facilitation Behavioural Assessment Tool. We recommend using the tool to support learning conversations for simulation faculty development in low‐resource settings.

Introduction

The worldwide expansion in healthcare simulation training [1] includes accelerated uptake in low‐resource settings [2]. Low‐resource settings are contexts with constrained financial or human resources where limitations in education and/or training, and underdeveloped infrastructure may impact healthcare negatively [3]. Low‐resource settings may include low‐income countries but can also exist within middle‐ and high‐income countries, for example in rural and remote locations [4]. A major barrier limiting the use of simulation in low‐resource settings is the paucity of educators with skills in the development and implementation of simulation activities [5]. Building skills in healthcare simulation facilitation is a complex task [6]. Skill development follows no uniform path with wide variations from informal self‐directed learning to comprehensive and structured training programmes [7, 8]. Learning within communities of practice [9, 10] is a well‐recognised approach to simulation faculty development [11]. The sociocultural learning theory posits that as learners become part of a community, they reflect on and adopt the practices, beliefs and values of that community [9].

The application of competency‐based frameworks within a community can support reflection on facilitation performance, promote facilitator skill development and optimise the quality of the simulation activities being conducted [12]. Until recently, no framework has specifically delineated the competencies underpinning effective facilitation practice in low‐resource settings [13]. The relative lack of contextually‐informed teaching and learning resources adds to the many challenges facing simulation facilitators in low‐resource settings [14].

This study is an extension of previous work conducted by central members of the Vital Anaesthesia Simulation Training (VAST) Community of Practice [15] to develop a core competency set for effective simulation facilitation in low‐resource settings [16]. Following a rigorous, international modified Delphi study, 32 core facilitation competencies were developed and arranged thematically into three performance categories, applying the Practice Development Triangle framework for expert debriefing practice [17]. The three performance categories are: techniques (the practical approach and micro‐skills of facilitation); artistry (the dynamic, creative and individualised nature of facilitation); and values (the fundamental principles that form the foundation of facilitation practice). This current study leverages the gains made in codifying facilitation competencies and integrates them into the Facilitation Behavioural Assessment Tool (online Supporting Information Appendix S1). Further detail on the design of the tool and pilot testing is provided in online Supporting Information Appendix S2.

The purpose of this study is to describe how the competency set developed in our Delphi study [16] has been incorporated into the Facilitation Behavioural Assessment Tool and to report reliability when scoring with this tool. The primary aim of this study was to determine inter‐rater reliability and internal consistency when using the tool. Our secondary aims were to determine if observers could discriminate between facilitation behaviours likely to have positive, negative or mixed influence on learning and the simulation environment, and to understand if previous facilitation experience influenced this determination.

Methods

This study was approved by the Human Research Ethics Committee, Curtin University, Perth, Australia. The requirement for written informed consent was waived, with completion of the online data collection inferring consent. This study adheres to reporting guidelines for healthcare simulation research [18]. As per the consensus statement on measures to promote equitable international partnerships in research [19], we have included a reflexivity statement (online Supporting Information Appendix S3).

We created three separate videos designed to depict facilitation behaviours likely to have a negative, positive or mixed impact on learning and the simulation environment. The clinical case involved management of hypotension following spinal anaesthesia for caesarean delivery. Intended learning objectives centred on recognition of hypotension, broadening the differential diagnosis and management of persistent hypotension post‐spinal anaesthesia. Videos were recorded in English, with accompanying English subtitles and lasted just under 20 min. Further details regarding video design are provided in online Supporting Information Appendix S2.

Power calculations for assessment of inter‐rater reliability using the confidence interval lower limit procedure [19] indicated at least 10 raters would be required to have sufficient power to detect a non‐zero relationship using many raters and three videos. This calculation assumed an expected intraclass correlation coefficient (ICC) of 0.80 and an 80% power. For internal consistency with Cronbach's α, assuming three items and 80% power comparing with a null hypothesis of zero and a hypothetical α level of 0.65, at least 24 raters were required [20]. To assess discrimination between levels of facilitation performance, assuming 80% power, a medium effect size explaining approximately 9% of the variance, correlations among repeated measures of r = 0.5 and homogeneity of variance and covariance, approximately 18 raters were required [21]. To assess difference in scoring between raters with varied previous simulation facilitation experience, assuming 80% power, a large effect size explaining approximately 25% of the variance and to detect a medium effect with 9% of the variance, 102 raters were required [21]. Therefore, to complete all planned data analysis, a sample size of 102 was required.

Volunteer response sampling [22] was used, with all healthcare providers who had previously completed the VAST Facilitator Course in English eligible for inclusion. An invitation sent to the VAST email database contained a participant information sheet and a link to a rater training module hosted on the VAST online learning platform. Demographic information was collected before a participant undertook the online training. Study participants had no previous knowledge of the videos and none had undergone previous formal training in the use of the tool. Participants were asked to self‐identify their previous simulation facilitation experience on entry to the study, with experience levels categorised as novice (< 10 scenarios); intermediate (10–50 scenarios); or experienced (> 50 scenarios).

All potential participants were pre‐enrolled in the online module, with a maximum of three automated reminder emails sent to encourage self‐enrolment and/or completion of uncompleted tasks at 2‐, 4‐ and 6‐weeks post enrolment. We anticipated diversity in the participant pool given the varied use of English as a first language and the wide geographical spread of potential participants. Training resources were designed to support asynchronous completion, at a time convenient to the participant at their own pace. The training included an orientation to the Facilitation Behavioural Assessment Tool and an explanation of its design and intended use. Throughout the training, participants were encouraged to develop a deep understanding of the performance categories, related competencies and behavioural anchors used to inform scoring. Apart from this self‐directed online content, there was no additional interaction with an expert trainer in learning how to use the tool. An artificial intelligence training time estimator built into the learning platform indicated the training module would take just under 2 h to review. Upon completion of the interactive content, study participants were presented with the videos to watch and score. Participants were advised to watch each video in its entirety before recording a score. Videos were presented in a random sequence, a process facilitated by the inbuilt question randomisation function of the platform. Other than offering thanks, no specific feedback or coaching regarding scoring was provided after submission of scores for each video.

To assess the inter‐rater reliability, intraclass correlations with 95%CIs using the two‐way random absolute agreement ICC2 and ICC2k methods for the total scores were calculated. The ICC2 method assesses absolute agreement for any single rater when there are multiple raters, and ICC2k assesses the reliability of a measure averaging across all k raters [23]. Koo and Li [24] suggest that ICC values < 0.5 show poor reliability; 0.5–0.75 are moderate; 0.75–0.90 are good; and values > 0.90 are excellent. To assess internal consistency, Cronbach's α with 95%CIs were calculated, including ‘techniques’, ‘artistry’ and ‘values’ as items. This produced three α statistics, one each for scoring of the positive, negative and mixed videos. Bland and Altman [25] suggest that values between 0.70 and 0.80 are satisfactory for comparing groups, with values of 0.90 desirable for clinical application.

To assess if raters using the tool could discriminate between different facilitation behaviours and if previous facilitation experience influences scoring, a single analysis was performed. A random intercepts 3 × 3 linear mixed model with video type as a within‐subjects predictor (positive, mixed and negative) and facilitator experience level as a between‐subjects predictor (novice, intermediate, experienced) was performed. As predictor variables were categorical, the statistical significance for main effects and interactions was calculated using analysis of variance; the Kenward‐Roger method [26] with type 2 sums of squares used [27]. All analysis was conducted in R version 4.4.1 (R Core Team, Vienna, Austria) using the psych and lme4 packages [28, 29]. A set of contrasts were conducted to probe the interaction effect. Of the maximum 36 possible pairwise comparisons that could be conducted in a 3 × 3 design, only nine comparisons were calculated to compare all three skill level groups when rating the positive, mixed and negative videos. These contrasts sufficed to probe the nature of the interaction. The p values for these tests were not adjusted for multiple testing, but familywise error rate was partially controlled for by running only nine of 36 tests. This approach was chosen to better balance type 1 and type 2 error rates, as a correction for 36 tests would inflate the false negative rate given the small sample size.

Results

In May 2024, an invitation email was sent to a total of 289 unique email addresses housed in the VAST database. By early September 2024, 104 (36%) participants had completed the training module and contributed to scoring on at least one video. There were 102 (35%) participants who provided scoring for the negative and mixed videos and 99 (34%) who scored the positive video. Participants were from 29 countries, spanning six continents (Fig. 1). Table 1 presents participant sociodemographic data and previous simulation facilitation experience. The inter‐rater reliability, when assessed with ICC2, was 0.73 (95%CI 0.66–0.79) and with ICC2k was 0.89 (95%CI 0.85–0.92). Cronbach's α was used to determine internal consistency; the Cronbach's α were 0.84 (95%CI 0.79–0.89), 0.84 (95%CI 0.78–0.88) and 0.91 (95%CI 0.87–0.93) for the positive, mixed and negative influence videos, respectively.

Figure 1.

Country of clinical practice for study participants. Numbers in brackets and dot size reflect the number of study participants from each country.

Table 1.

Study participant characteristics. Values are number (proportion).

	n = 104
Age; y
20–29	5 (5%)
30–39	40 (38%)
40–49	29 (28%)
50–59	19 (18%)
> 60	11 (11%)
Gender
Female	63 (61%)
Male	40 (38%)
Prefer to not disclose	1 (1%)
Profession
Physician anaesthesia provider	76 (73%)
Nurse	12 (11%)
Medical officer	5 (5%)
Non‐physician anaesthesia provider	4 (4%)
Non‐clinical role	3 (3%)
Obstetrician/gynaecologist	2 (2%)
Midwife	1 (1%)
Surgeon	1 (1%)
Economic status of country of clinical practice
Low‐ and middle‐income	63 (61%)
High‐income	41 (39%)
Previous facilitation experience
Novice (< 10 sessions)	49 (47%)
Intermediate (10–50 sessions)	36 (35%)
Experienced (> 50 sessions)	19 (18%)

Participants of all experience levels were able to distinguish meaningfully between the positive, mixed and negative videos. There was a main effect of video type (F(2200) = 183.85, p < 0.001). The main effect for video type tests the null hypothesis that the three means for video type are equal; thus, these results suggest the three means differ from each other, after collapsing across facilitator experience level. To explore this further, we examined mean scores and pairwise comparisons. When collapsing all participants into one analysis, the mean total score was 8.54 (95%CI 8.19–8.90) for the positive video, 6.25 (95%CI 5.91–6.60) for the mixed video and 4.17 (95%CI 3.82–4.51) for the negative video. All pairwise comparisons between these three means were statistically significant at p < 0.001 (Table 2). The regression output of the full linear mixed model, as well as means and confidence intervals for all nine conditions are available in online Supporting Information Tables S1 and S2.

Table 2.

Comparison of mean rater scores between the three videos, grouping raters by simulation facilitator experience level (novice < 10 sessions; intermediate 10–50 sessions; and experienced > 50 sessions of simulation training delivered).

Comparison	Mean difference	95%CI	p value
Positive video
Novice vs. intermediate	‐0.11	‐0.84–0.61	0.755
Novice vs. experienced	‐0.23	‐1.13–0.67	0.610
Intermediate vs. experienced	‐0.12	‐1.07–0.83	0.805
Mixed video
Novice vs. intermediate	1.09	0.37–1.82	0.003
Novice vs. experienced	1.08	0.20–1.97	0.016
Intermediate vs. experienced	‐0.01	‐0.94–0.92	0.986
Negative video
Novice vs. intermediate	0.99	0.26–1.17	0.008
Novice vs. experienced	0.90	0.02–1.79	0.046
Intermediate vs. experienced	‐0.08	‐1.02–0.85	0.858

Mean difference in total scoring using the Facilitation Behavioural Assessment Tool with total score ranging from a minimum of 3 to a maximum of 9. The 95%CI is calculated around the mean difference. These tests represent planned contrasts comparing means in groups. The study design is a 3 × 3 design with video type (positive, mixed, negative) and experience (novice, intermediate, experienced) as the x variables, and the y variable being total scores on the Facilitation Behavioural Assessment Tool. As an example, the novice vs. intermediate mean difference for the positive video is ‐0.11. This means that the mean in the novice group minus the mean in the intermediate group for positive videos only was ‐0.11, with participants in the novice group having a smaller mean. The p values test the null hypotheses that the mean difference equals zero. The p values were not adjusted for familywise error in this table.

Novice facilitators scored facilitation behaviours higher for the mixed and negative videos compared with participants with intermediate and high levels of experience. Participants with intermediate levels of facilitation experience did not differ significantly in their scoring compared with more highly experienced raters. We found a significant difference between mean scores grouped by facilitator experience (F(2101) = 4.62, p = 0.012 (online Supporting Information Appendix S4)). When examining the means and pairwise comparisons (Table 2), the pattern was seen more clearly: participants with novice levels of previous facilitation (mean 6.73, 95%CI 6.44–7.03) rated the mixed video more highly than participants with high levels of previous facilitation experience (mean 6.15, 95%CI 5.67–6.64, p = 0.018), but did not differ significantly from raters with intermediate levels of experience (mean 8.54, 95%CI 5.73–6.43, p = 0.089). Moreover, participants with intermediate and high previous facilitation experience did not differ from each other (p = 0.805). These main effects were qualified by a two‐way interaction (F(4200) = 2.43, p = 0.049). Figure 2 shows means for total scores split by level of facilitation experience and video type. Further details on the mean comparisons and regression output can be found in online Supporting Information Appendix S4.

Figure 2.

Dot and violin plots with means for total scores across split by level of facilitation experience and video type. Red dots, mean values with the width of the violin representing the distribution of the data; green dots, individual observations for participants with novice previous facilitation experience; orange dots, participants with intermittent previous facilitation experience; purple dots, participants with extensive previous facilitation experience. Left panel, positive videos; middle panel, mixed videos; right panel, negative videos.

Discussion

To our knowledge, this study represents the first step in building a validity argument [30] supporting the use of the Facilitation Behavioural Assessment Tool for faculty development in low‐resource settings. We found that the tool has suitable reliability when used by healthcare providers following a short, online rater training module. Our study supports the use of this tool for making observations and guiding assessment of simulation facilitation [30]. This is important because high‐quality simulation facilitation can optimise the participant experience in simulation and enhance transfer of learning into clinical practice. To date, those conducting simulation in low‐resource settings have not had the benefit of a contextually‐informed facilitation assessment tool [13]. Further work is required to explore the optimal application of the Facilitation Behavioural Assessment Tool, including investigating the extrapolation and implications of its use [31]. We recommend that the Facilitation Behavioural Assessment Tool be used predominantly for reflective learning conversations on facilitation performance in a structured, theoretically informed and psychologically safe manner [32]. Ideally, these conversations should be integrated into an intentional programme of peer coaching to enhance debriefing skills [12].

All raters, regardless of previous facilitation experience, were also able to use the Facilitation Behavioural Assessment Tool to discriminate meaningfully between videos representing facilitation behaviours that would likely lead to positive, negative or mixed influence on learning and the simulation environment. The inter‐rater reliability when using the Facilitation Behavioural Assessment Tool was 0.73 (95%CI 0.66–0.79) for a single rater. If multiple raters scores are averaged, the inter‐rater reliability of the Facilitation Behavioural Assessment Tool increased to 0.89 (95%CI 0.85–0.92). The internal consistency when using the Facilitation Behavioural Assessment Tool was 0.84 for rating of positive and mixed videos and 0.91 for the negative video. The statistically significant sub‐analysis indicated that novice facilitators scored slightly higher for mixed and negative videos.

The benchmark status of the Debriefing Assessment for Simulation in Healthcare (DASH) [33] tool in the simulation community inspired our approach to the evaluation of the tool. In our study, we were able to show inter‐rater reliability comparable with the DASH [33] at 0.74 for a single rater, and exceeding it for multiple raters. Likewise, our results for internal consistency are comparable with the single calculation of internal consistency for the DASH at 0.89 [33]. This finding is important as we intentionally avoided the expert in‐person synchronous instruction that occurred as part of the DASH rater training [33], whereby raters participated in interactive discussions, being given feedback on scoring across two videos before the calculation of inter‐rater reliability on the scoring of a third and final video. The sub‐analysis showing novice raters scored higher than more experienced raters is a consistent finding for novice raters using other assessment tools [34] and may indicate that increased exposure to simulation facilitation modifies the perspective of observers.

Potential barriers to the uptake of simulation in low‐resource settings have been categorised into three broad groups: academic; resource‐based; and professional factors [5]. Under each factor there are nine themes including deficits in the curriculum and facilitator skills; learner factors; costs; infrastructure and personnel limitations; competing priorities; lack of leadership; and poor organisational support [5]. The value of this work relates to its ability to address two of these themes: the lack of contextually‐informed learning resources; and a shortage of skilled simulation facilitators.

Strengths of this study relate to its accessible, pragmatic approach to facilitation assessment, grounded in the practicalities of simulation faculty development in low‐resource settings [15, 16, 17, 35]. The design of our tool (online Supporting Information Appendix S2) with a three‐point rating scale inherently challenges the ability to show inter‐rater reliability, compared with those with higher points of discrimination [36]. We intentionally chose a rating scale with fewer points of discrimination, prioritising simplicity in use; our ability to show similarly favourable psychometric properties to the seven‐point rating scale in the DASH [33] is a major strength. Furthermore, the reliability and internal consistency has been shown after only a short episode of online self‐directed rater training.

The Facilitation Behavioural Assessment Tool is a resource directly informed by facilitation practice in low‐resource settings. In their review of simulation uptake in low‐resource settings, Ismail et al. identify that there is a strong desire from educators to develop skills in simulation‐based education [5]. Pairing this enthusiasm with structured, high‐quality faculty development initiatives that provide varied learning experiences, mentorship and opportunities for self‐reflection will lead to enhanced capacity for high‐quality simulation programmes [1, 5, 6, 7, 8, 11, 12, 37].

A limitation of our study relates to the potential of sampling as a source of bias. Our study included participants with previous involvement in VAST's programmes, potentially influencing scoring as they may already have been accustomed to facilitation behaviours categorised by the tool. Our study had a response rate comparable with the average response rate for online surveys [38], which is notable given the time commitment required by participants compared with a simple online survey. Notwithstanding, it is difficult to discern the effect that contribution by non‐responders may have had on the results. The generalisability of the results to our broader simulation community of practice is, however, supported by the diversity of study participants. An additional limitation is that our study was not powered to detect a main effect of sub‐groups within grouping based on previous facilitation experience. Further investigation with greater participant numbers is required to explore if, for example, those with novice levels of previous facilitation experience from low‐ and middle‐income countries score differently compared with novices from high‐income countries. Videos were recorded and subtitled in English only, potentially influencing the scoring of those with varied English proficiency. Despite recruiting 104 participants in total, only receiving 99 scores on the positive video compared with the a priori sample size calculation of 102 is a further limitation. Finally, the data presented in this study relates specifically to the use of the tool in the specific circumstances detailed in this research project. As highlighted above, there is impetus for further exploration of the application of this tool in wider contexts.

In conclusion, this study supports the hypothesis that observers using the Facilitation Behavioural Assessment Tool have good inter‐rater reliability, high internal consistency and can discriminate reliably between different facilitation behaviours that are likely to have either a positive, mixed or negative influence on learning and the simulation environment. As with any assessment tool, rater training is an essential step in promoting the reliability of its use, and our study shows that reliability can be achieved following highly accessible and pragmatic rater training. We recommend that the Facilitation Behavioural Assessment Tool is used to promote reflection on facilitation performance, ultimately promoting high‐quality simulation activities across the globe.

Supporting information

Appendix S1. Facilitation behavioural assessment tool.

Appendix S2. Facilitation behavioural assessment tool design.

Appendix S3. Reflexivity statement.

Table S1. Regression output for linear mixed model.

Table S2. Means, standard errors and confidence intervals for all conditions.