Simulation-based evaluation of anaesthesia residents: optimising resource use in a competency-based assessment framework

Abstract

Introduction

Simulation training in anaesthesiology bridges the gap between theory and practice by allowing trainees to engage in high-stakes clinical training without jeopardising patient safety. However, implementing simulation-based assessments within an academic programme is highly resource intensive, and the optimal number of scenarios and faculty required for accurate competency-based assessment remains to be determined. Using a generalisability study methodology, we examine the structure of simulation-based assessment in regard to the minimal number of scenarios and faculty assessors required for optimal competency-based assessments.

Methods

Seventeen anaesthesiology residents each performed four simulations which were assessed by two expert raters. Generalisability analysis (G-analysis) was used to estimate the extent of variance attributable to (1) the scenarios, (2) the assessors and (3) the participants. The D-coefficient and the G-coefficient were used to determine accuracy targets and to predict the impact of adjusting the number of scenarios or faculty assessors.

Results

We showed that multivariate G-analysis can be used to estimate the number of simulations and raters required to optimise assessment. In this study, the optimal balance was obtained when four scenarios were assessed by two simulation experts.

Conclusion

Simulation-based assessment is becoming an increasingly important tool for assessing the competency of medical residents in conjunction with other assessment methods. G-analysis can be used to assist in planning for optimal resource use and cost-efficacy.

Introduction

Simulation in anaesthesiology training was widespread by the early 2000s, fuelled by evidence that emerged in the mid-1980s that simulation training could be an effective teaching tool in medicine. ^1–10 The early adoption of simulation training in anaesthesiology was spurred on by the need to provide training for high-stakes, but low-frequency, clinical situations. ^{1 2 7 11} This is of particular concern in anaesthesiology, where powerful, rapidly acting drugs, complex devices and invasive procedures are often required in urgent care situations, ¹² and where physician error can be a significant contributor to adverse outcomes. ^{1 2} Today, simulations are used for both competency-based training and assessment, ^{2 6 7} but one of the largest barriers to the further expansion of simulation is the associated monetary costs and significant resource demands. ^{3 13 14} High-fidelity simulation is expensive because it often involves significant technological resources and demands staff (faculty/facilitators) time to plan, execute and evaluate simulations. The most common barriers identified in a recent nationwide survey regarding the use of simulation in preparation of anaesthesiology trainees included lack of time, lack of faculty expertise and lack of funding. ¹⁵

While evidence suggests that simulation-based assessments are effective for measuring residents’ competencies, ¹⁶ the number of simulation-based assessments needed for accurate competency-based assessment remains to be determined. This question is of particular importance for smaller academic institutions that have established regimens of resident assessment and are considering the introduction of simulation-based training and assessment into their competency-based training programme.

For the purposes of our training programme, it was essential to determine the residents’ capabilities to ensure the provision of safe independent care (often during on-call periods). Simulation-based assessments in the form of objective structured clinical examinations are key tools used by the Royal College of Physicians and Surgeons of Canada and other accreditation councils across multiple disciplines to confirm readiness for independent practice. ^17–21

Given the limited exposure to rare acute events, a simulation-based assessment was introduced into the early stages of our training programme with a focus to identify how residents applied the principle of crisis resource management to distinct clinical scenarios. Simulation is a valuable educational tool and is potentially ideally suited to address this identified need in our residency programme. However, high-fidelity simulation currently requires significant human resources, including technologists, standardised actors, expert assessors and simulation faculty. In an era of limited financial and human resources, the ability to deliver simulation in the most efficient and cost-effective manner is necessary for its sustained use within our educational programme.

In this paper, we address how statistical modelling can be used to investigate the optimal number of scenarios and faculty needed to achieve a reliable ‘readiness for practice’ score.

Our primary hypothesis was that generalisability analysis (G-analysis) and sensitivity testing can be used to estimate the influence of measurement error on assessment outcomes and the number of assessments and raters needed for accurate assessment.

G-analysis is an extension of classical test theory (CTT), which is a set of research principles that form the theoretical premise of reliability testing in observational research. CTT presupposes that any observed event is composed of an actual event plus a measurement error. G-analysis isolates sources of measurement error and estimates its impact on the observational data. ²² The sources that contribute to measurement error are referred to as independent ‘facet’ variables. The variance attributable to participants is included in G-analysis, and in a world where there was no measurement error, participants would account for all the variance in the observed data. ^23–26 The potential application of G-analysis within educational practice was previously reviewed in depth by Bloch and Norman. ²⁷

Methods

Participants

This prospective cohort study draws data from residents entering their first year of training in the Queen’s University Anaesthesiology Postgraduate Training programme over a 3-year period. Residents completed a 4-week anaesthesiology rotation in the first year of residency and another 8-week anaesthesiology rotation immediately prior to the simulation. Anaesthesiology training in Ontario adheres to the residency model of postgraduate apprenticeship. Under the residency model, the quantity and variety of clinical cases in which residents are exposed is tied to patient volume and pathology. To enhance learning opportunities, simulation-based education was introduced to the Queen’s University anaesthesiology residency curriculum in 2006.

With consent, data were collected from 17 anaesthesiology residents who each performed four structured simulations. Three Postgraduate Year 3 (PGY3) residents who were previously approved for (and were already effectively performing) independent call were used as performance comparators (n=3) to ensure the assessment rubrics did not have unrealistic expectations. All the ther subjects (n=14) were at equivalent stages of residency training and were being considered for initiation of independent call responsibilities. Six residents were male and 11 residents were female. The quantitative component of the analysis was conducted using the Statistical Package for the Social Sciences (SPSS) V.24.

Faculty assessors

Scenarios were independently evaluated by two staff anaesthesiologists well versed in simulation. Both faculty have been extensively involved in simulation-based education (>5 years) and formal assessment of residents at the national level. All simulations were independently evaluated in real time using a descriptive rubric to assess competency across six dimensions of practice, including evaluation, management plan, differential diagnosis, emergency response, critical features addressed and communication skills. The resident’s performance across these domains informed a final global rating score derived from a 9-point scale modified from formal assessment tools used in oral anaesthesiology examinations(http://www.royalcollege.ca/rcsite/documents/ibd/anesthesiology_examformat_e). ²⁸ All evaluations were completed by the faculty assessors prior to the debriefing session for each scenario.

Scenarios

Each resident performed four emergency simulation scenarios that were developed through internal consensus of two simulation faculty and the programme director to reflect common on-call responsibilities as outlined in the defined competencies (online supplementary appendix 1). Scenario topics were (1) an obstetrical emergency, (2) management of acute pain, (3) an intraoperative emergency and (4) resuscitation in a satellite location. The components of an ideal response to the evaluation and management of each scenario were outlined in a formal checklist developed by the simulation faculty and the programme director. Specific actions that were deemed critical to the safe and timely management of patients were highlighted in the checklist as features critical to each scenario. Three anaesthesia residents already performing independent on-call duties were included to ensure validity of both the content of the scenarios and the assessment instruments.

The simulation scenarios involved a high-tech simulation manikin (SimMan 3G, Laerdal), a staged high-fidelity treatment room and support staff. The support staff were provided with scripts for responding to resident inquires. They also received additional audio prompting from the simulation faculty to standardise the scenario. All simulations were videotaped and available for review with the residents during the postsimulation debriefing.

Data analysis

We estimated the reliability of our global assessment instrument using a G-analysis of the extent of variance attributable to three unique variables (aka facets): (1) the scenarios, (2) the assessors and (3) the participants. The variance in the simulation scores attributable to resident performance is desirable because it means that the variance in final scores is emanating from the residents and not from extraneous sources. ^23–25 The variance attributed to scenarios would indicate that resident performance differed by scenario and that entrustment of a resident is dependent on the case that is presented. Variance between assessors would indicate that resident scores are dependent on who is assessing them. Variance attributable to the interaction between each of these three facets is also considered. The G-coefficient is the ratio comparing the variance emanating from participants themselves with the variance emanating from the interaction between participants and the other variables.

The D-coefficient is the ratio comparing the variance emanating from residents with the variance emanating from all sources of measurement error, also known as the absolute measurement error. The D-coefficient complements the G-coefficient by indicating whether the results of the study are dependable or if their dependability has been eroded by measurement error. Generalisability study (G-study) analysis was done on SPSS using custom syntax recommended by Mushquash and O’Connor. ²⁹

Sample sizes and explanatory power

It is difficult to definitively establish the number of residents and/or the number of facets needed to empower a G-study. ³⁰ Small sample sizes are a reality faced by many medical residency programmes collecting and analysing resident assessment data. The number of medical residents completing a rotation in anaesthesiology is relatively small at each institution (four to five per year at our institution), and data collected at institutions over time are affected by changes in curriculum and assessment. Fourteen subjects and three performance comparators, for a total of 17 subjects, represent complete capture of data available during 3 years at our academic centre. However, the sample size (n=2×4×17=136 scores) available for analysis in testing our hypothesis is larger because it compares residents’ (17) individual simulation scores (4) across raters (2). The G-analysis treats every simulation assessment score as the unique result of several discrete factors, the independent facet variables.

Results

The simulation-based assessments were successfully implemented within our resident training programme in anaesthesiology with all residents (n=17) completing the four structured scenarios. When using the global rating scale, an average rating of 70 or greater was defined as a resident who was ready for independent call responsibilities (n=8, 47%), whereas a global rating between 65 and 69 was deemed to represent a borderline performance (n=6, 35%). The distribution of mean global rating scores for each resident after completion of the four scenarios is shown in figure 1.

Figure 1.

Distribution of mean global rating scores by participant. Shaded region (65–70) represents borderline performance (n=6).

Estimates of the variance emanating from the G-study facets are presented in table 1. The results indicate that 47% of the variance in the data comes from the residents in the study and that 37% of the variance can be attributed to an interaction effect between residents and the scenarios. This coincides with the descriptive data in table 2 showing that average scores between scenarios varied little, but most notably between scenarios 3 (intraoperative emergency) and 4 (resuscitation in a satellite location). The remaining 16% of the variance was captured by the error term and is unaccounted for by the facets. Table 1 shows that the D-coefficients and G-coefficients for this study are both above 0.80 (G=0.81 and D=0.81), indicating high degrees of both generalisability and reliability. ^{27 31}

Table 1.

Reliability and measurement error

G-study facet variables	Sig.	Est.	% variance		n
Person (p)	*	35.61	46.63	σ2(p)	17
Rater (r)		0	0	σ2(r)	2
Scenario (o)		0	0	σ2(o)	4
pXr		0	0	σ2(pr)	34
pXo	†	28.14	36.84	σ2(po)	68
rXo		0.63	0.82	σ2(ro)	8
pXrXo, +error		12.00	15.71	σ2(pro)	136
Total variance		76.37
Universal score variance, σ2(τ)=σ2(p)		35.61
Relative error variance, σ2(δ)=(σ2(pr)/nr)+(σ2(po)/no)+(σ2(ptr)/nrXno)		8.53
Absolute error variance (everything but σ2(p)/np)		8.61
Generalisability coefficient, Ep2=σ2(τ)/(σ2(τ)+σ2(δ))		0.81
Dependability coefficient, Ep2=σ2(τ)/(σ2(τ)+σ2(Δ))		0.81

Bold faced values were used to emphasize the variance from person (learner) and interaction between person (p) and scenario (o) as the two largest facets accounting for variation observed. The G and D coefficients are highlighted as the main measure in the G study.

p <0.05

†p <0.01

est., estimate; G-study, generalisability sudy; sig., significance.

Table 2.

Descriptive statistics for faculty assessors

Rater	Scenario 1	Scenario 2	Scenario 3	Scenario 4
	Obstetrics	APMS	OR Bradycardia	Cath Lab Intubation
1	70.00	72.06	70.59	67.35
2	69.71	68.82	70.59	68.24
Average	69.85	70.44	70.59	67.79
Difference	−0.29	−3.24	0.00	0.88

APMS, Acute Pain Management Service; OR, Operating Room.

Parametric testing can help determine the robustness of the D-coefficient and the G-coefficient, as well as help determine achievable accuracy targets, such as 0.80. Increasing raters from two to three increases this study’s expected G-coefficient from 0.807 to 0.816. Changing the number of scenarios has a greater effect on the G-coefficient; increasing the number of scenarios from four to five and from five to six increases the coefficient from 0.807 to 0.839 and 0.862 respectively, but at a decreasing rate of return. Table 3 shows that the greatest gains in the G-coefficient accrue from the first few scenarios. Running more than four training scenarios, the data indicate mean marginal gains in assessment accuracy for additional cost.

Table 3.

Reliability parametric testing of assessment accuracy

	Change in number of scenarios
	Subtract 2	Subtract 1	Current	Add 1	Add 2
Raters	2	2	2	2	2
Change in number of scenarios	2	3	4	5	6
Generalisability coefficient	0.676	0.758	0.807	0.839	0.862
Change		0.082	0.049	0.032	0.023
Dependability coefficient	0.674	0.756	0.805	0.838	0.861
Change		0.082	0.049	0.033	0.023
			Current	Add 1	Add 2
Raters			2	3	4
Scenarios			4	4	4
Generalisability Coefficient			0.807	0.816	0.821
Change			0.009	0.005
Dependability Coefficient			0.805	0.815	0.820
Change			0.010	0.005

Bold faced type indicates the current practice with regards to number of raters and scenarios used in the study. The other values are deviations from the number of scenarios or raters used.

Figure 2 graphs the G-coefficient and the marginal change in G-coefficient as the number of scenarios increases from one through six while holding the number of raters fixed at 2. The decreasing marginal gains in the G-coefficient for each incremental increase in the number of scenarios suggests that there may be an optimal balance at which the G-coefficient is maximised at the least cost. Figure 2 shows that the G-coefficient crosses the 0.80 threshold between three and four scenarios, which is where the marginal change in G (plotted using the dotted line) begins to level off.

Figure 2.

Representation of the change in G-coefficient with the increasing number of scenarios. The highlighted vertical bar suggests an efficient administrative ‘sweet spot’ that maximises the G-coefficient for the least resource use.

Discussion

The results of this study lend new insight into the value of generalisability theory for balancing the number of scenarios and assessors required for optimal simulation-based assessment. Our study confirms that G-analysis and sensitivity testing can be successfully used to estimate the influence of measurement error on assessment outcomes and the number of assessments and raters needed for accurate assessment. In particular, we demonstrated that four scenarios and two expert raters were adequate to ensure assessment reliability within our anaesthesia training programme. A parametric analysis of the G-study indicates that additional simulations and raters only improve reliability marginally and at a decreasing rate. The interaction between scenarios and residents by contrast tended to be a source of relatively high error variance (measurement error). The parametric analysis shows that increasing the number of simulations can average out this scenario–interaction variance, decreasing its significance.

The findings of the G-study parameters are likely to be of interest to medical educators and programme administrators looking to balance assessment quality with associated costs of simulation. While several simulations are required for accurate assessment (in this case four), administrators can be confident that assessment quality can be retained with as few as two expert raters. What can be inferred from the G-analysis is that there are an optimal number of simulations and raters needed to achieve an acceptable level of assessment reliability (eg, coefficient of greater than 0.80 or 0.85), and that little benefit is gained from adding simulation sessions or raters, which therefore allows for cost savings.

This study was developed as our local simulation group contemplated introducing simulation-based assessments into our residency training programme. As with other specialty programmes, the study population was limited by annual enrolment in our training programme (n=5). The relatively small sample size is not an uncommon problem encountered by graduate residency programmes in anaesthesia, and is a recognised limitation of the current study. Our decision to include the data from the three performance comparators in the G-analysis has the potential to influence attribution to various facets. The application of G-study parameters to optimise the application of limited resources to simulation-based assessment is an area that could be further explored in future multicentre educational endeavours coordinated at the national level. ³² With the current transition to a competency-based framework for postgraduate medical education in Canada, the need for effective simulation-based assessments is anticipated to expand. We believe our application of G-study principles can assist in balancing the distribution of limited simulation resources for maximal efficacy.

Conclusion

Multivariate G-analysis and sensitivity testing were used successfully to estimate the optimal number of simulation scenarios and raters required for accurate simulation assessment. In this study, the optimal balance was obtained when four separate scenarios were assessed by two expert simulation faculty. This finding is likely to be of interest to medical educators and programme administrators involved in simulation-based assessment because it may allow for optimal efficacy with minimal associated costs.