Validity and reliability of simulation-based assessment tools for European Training Requirement-defined competencies in anaesthesiology: a two-centre observational pilot study

Summary

Despite ongoing efforts to harmonise postgraduate anaesthesiology training across Europe, significant variability persists among national programmes, and a unified certification process is lacking. Simulation is widely used in postgraduate anaesthesiology training, and several validated tools are available to assess both technical and nontechnical skills. However, use of such tools for summative assessments in specialist certification has not yet been established. This prospective, observational pilot study aimed to evaluate the validity and inter-rater reliability of two scenario-specific checklists, a global rating scale (GRS), and the Anaesthetists’ Non-Technical Skills (ANTS) tool in assessing competencies defined by the European postgraduate anaesthesiology curriculum. The study was conducted between May and September 2022 at two university hospitals in Germany and Spain. A total of 43 (23 third-year and 20 fourth-year) anaesthesiology trainees completed a knowledge test and managed two simulated perioperative emergencies. Performances were video-recorded and independently assessed by two blinded raters. Fourth-year trainees scored significantly higher than third-year trainees across all tools, providing validity evidence. Inter-rater reliability was moderate for Scenario 1 (intraclass correlation coefficient [ICC] range: 0.50 to 0.67) and moderate-to-good for Scenario 2 (ICC range: 0.67 to 0.76). These findings suggest that simulation-based assessments using structured tools can validly and reliably evaluate technical and nontechnical competencies outlined in the European anaesthesiology training curriculum. The results support the integration of such assessments into standardised summative evaluation frameworks in European postgraduate education.

Editor’s key points.

• •Significant international variability exists between anaesthesiology training programmes and certification processes. Although simulation is widely used in postgraduate training, its use for assessments in specialist certification has not yet been established.
• •The authors performed a prospective, observational pilot study in two university hospitals to evaluate the validity and inter-rater reliability of simulation in assessing competencies as defined by the European postgraduate anaesthesiology curriculum.
• •This simulation study supports the validity and reliability of simulation-based assessments for evaluating European Training Requirement-defined competencies in advanced anaesthesiology trainees across two European centres by showing that structured tools can validly and reliably evaluate technical and nontechnical competencies.
• •Future studies should include comparisons with existing assessment tools and expansion to broader cohorts of trainees and diverse healthcare systems to investigate their feasibility and applicability for postgraduate anaesthesiology training.

The European Union of Medical Specialists (UEMS) is the representative organisation for national associations of medical specialists across Europe. Each recognised medical specialty within the European Union (EU) is represented by a European board, which is responsible for establishing the European Training Requirements (ETR) and organising the European specialty examination. The ETR defines standards for postgraduate medical specialist training. In anaesthesiology, the European Board of Anaesthesiology (EBA) and the European Society of Anaesthesia and Intensive Care (ESAIC) have collaborated to harmonise postgraduate medical training across Europe.1, 2, 3 This collaboration has led to the creation of the ETR in anaesthesiology³ and the establishment of the European Diploma in Anaesthesiology and Intensive Care (EDAIC) examination.^4,5 Despite these efforts, significant differences remain among European training programmes^1,6, 7, 8 and no unified certification process for anaesthesiology specialists has yet been established.^1,5,6,8

Simulation is widely used globally in anaesthesiology training programmes for educational purposes and formative assessments.^1,6,9,10 Several validated tools exist for assessing technical and nontechnical skills during simulated scenarios.11, 12, 13, 14 Additionally, simulation has been increasingly integrated into summative assessments for anaesthesiology trainees in the USA15, 16, 17 and Canada,^18,19 and it has also become an important component of the oral certification examination administered by the Israeli Board of Anaesthesiology.²⁰ However, at the European level, only the Objective Structured Clinical Examinations (OSCE) component of the Primary Fellowship of the Royal College of Anaesthetists (FRCA) examination currently includes a simulation-based skills assessment. Neither the Final FRCA nor the EDAIC Part II examinations incorporate such assessments.^10,21, 22, 23 Moreover, simulation within European anaesthesiology training programmes primarily serves educational purposes, such as formative assessment and competency-based education.²⁴ The role of simulation in summative assessment for specialist certification in Europe remains a subject of ongoing debate.¹⁴

To date, no evaluation tools have been systematically validated for summative assessments aligned with ETR-defined competencies across different national training systems in Europe. This underscores a critical need to validate structured, reliable, and ETR-aligned assessment instruments that can be consistently implemented across postgraduate anaesthesiology training programmes throughout Europe.

This pilot study aimed to evaluate the validity and reliability of two scenario-specific checklists, a GRS and the Anaesthetists' Non-Technical Skills (ANTS) system as a part of a simulation-based assessment of ETR competencies in advanced anaesthesiology trainees in Germany and Spain.

Methods

Ethics approval

Ethical approval was granted by the Ethics Committee of the Medical Faculty, Heidelberg University, Heidelberg, Germany, on December 6, 2021 (S-879/2021), and by the Research Ethics Committee for Medicines, Hospital Clínic de Barcelona, Barcelona, Spain, on January 13, 2022 (HCB/2021/1334). Written informed consent was obtained from all participants.

Study design and setting

This prospective, observational pilot study was conducted at the simulation centres of two European university hospitals. Participants were enrolled in simulation sessions conducted between May and September 2022. Recorded performances were evaluated by two raters between October 2022 and March 2023. This manuscript adheres to the applicable STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) guidelines²⁵ and the extensions for simulation research²⁶ (Supplementary material 1, Supplementary Tables 1 and 2) .

Participants

A purposive sampling strategy was used. Given the differences in training duration between Spain (4 yr) and Germany (5 yr),⁸ third- and fourth-yr anaesthesiology trainees were recruited from both centres to ensure consistency across both countries.

Study procedures

Participants completed a demographic questionnaire and a knowledge test, before undertaking two simulated perioperative anaesthesia scenarios. Each participant's performance was recorded for subsequent evaluation by two independent assessors. At the end of the session, participants were asked to complete a post-simulation questionnaire assessing the scenario fidelity to real-life situations.

Simulation scenarios and assessment tools

To assess theoretical knowledge, a test based on Part I of the EDAIC was developed (see Supplementary material 2 for details of the development process, and Supplementary material 3 for the complete knowledge test).

To evaluate clinical skills, expert anaesthesiologists from the research team designed two standardised 15-min high-fidelity perioperative simulation scenarios, focusing on competencies selected from the ETR for anaesthesiology,²⁷ with an emphasis on behaviours observable in a simulated environment. The two scenarios were: (1) anaphylaxis with angioedema and unexpected difficult airway; and (2) local anaesthetic systemic toxicity with generalised seizure and cardiac arrest (see Supplementary material 4 for list of ETR-defined competencies and complete scenario scripts).

A structured two-part assessment tool was developed: the Structured Assessment of Clinical Skills in Simulation (SACSS). It consisted of a scenario-specific checklist (SACSS-checklist) and a GRS (SACSS-GRS). The checklist items were scored dichotomously (0, not performed or incomplete; 1, performed correctly), with final scores reported as percentages. The SACSS-GRS included 10 items rated on a five-point Likert scale; the final score was calculated as the mean across items and expressed as a percentage (see Supplementary material 2 for development details and Supplementary material 5 for the complete SACSS tool).

Nontechnical skills were evaluated using the validated ANTS tool²⁸ (see Supplementary material 2 for details).

Simulation environment

Scenarios were conducted in standardised simulated operating theatres at both centres, using different simulation manikins: a Human Patient Simulator® (Medical Education Technologies, Inc.®, Sarasota, FL, USA) in Germany, and a SimMan® (Laerdal Medical, Stavanger, Norway) in Spain. Both scenarios followed scripted timelines for vital signs progression. Two confederates acted as the anaesthesia nurse and the surgical nurse throughout each scenario, with a third confederate playing the role of an anaesthesia consultant to assist participants at scenario conclusion. Simulation sessions were conducted in the local language of each centre (see Supplementary material 2, which details scenario standardisation).

Training of video evaluators and rating procedures

One research team member and one external collaborator served as video evaluators; both were bilingual in German and Spanish, certified specialists in anaesthesiology, EDAIC diplomates and experienced educators in simulation-based training. After reviewing background literature,²⁹ they received training on the use of the rating tools before assessments (see Supplementary material 2 for more details). All encounters were independently evaluated by both raters, blinded to each other’s assessments, using a standard media player allowing multiple viewings. Checklist actions were scored in real time and considered completed if performed by participants or confederates under their direction. The SACSS-GRS and ANTS scores were assigned after viewing the entire recording. To minimise bias owing to the chronological viewing of the recordings, a randomly generated order of the viewings was assigned to each evaluator using an online random number generator (www.randomizer.org).

Data management

Videos were initially recorded onto simulation facility computers, transferred to the first evaluator's computer at the end of each session, and immediately encrypted using FileVault for Mac (macOS Monterey, version 12.7.2; Apple Inc., Cupertino, CA, USA). Rater scores were recorded on anonymised paper forms, scanned, encrypted, and later transcribed into an anonymised Excel database, version 16.16.27 (Microsoft Corporation, Redmond, WA, USA). Figure 1 illustrates the sequence of interventions and data collection.

Fig 1.

Flowchart illustrating study procedures, assessment processes, and data collection. ANTS, Anaesthetists’ Non-Technical Skills; SACSS, Structured Assessment of Clinical Skills in Simulation.

Evaluation of validity and reliability of the assessment instruments

Following Messick's contemporary framework, validity evidence was collected from three sources: content, relationships with other variables, and internal structure.^11,30 The simulation scenarios and SACSS tools were designed by experts to assess ETR-defined competencies, supporting evidence based on content.^11,30 Evidence based on relationships with other variables was examined by: (1) comparing scores between more (fourth-yr) and less experienced (third-yr) trainees; and (2) analysing expected convergent and divergent correlations between instruments assessing related or distinct constructs (e.g. theoretical knowledge, clinical skills, nontechnical skills). Internal structure evidence was evaluated by: (1) examining inter-rater reliability; and (2) analysing cross-scenario consistency (i.e. stability of participant scores across the two scenarios).^11,30

Sample size

The sample size of 40 trainees was predetermined based on logistical and financial limitations.

Statistical analysis

Descriptive statistics were used to summarise questionnaire data. Comparisons between more experienced (fourth-yr) and less experienced (third-yr) trainees were made using the Mann–Whitney U-test. Average scores from both evaluators were used in SACSS and ANTS comparisons. Linear correlations between different assessments were explored using Pearson correlation coefficients. Inter-rater reliability was assessed using global agreement and prevalence-adjusted and bias-adjusted Kappa (PABAK)³¹ for each item of the SACSS-checklist, and intraclass correlation coefficient (ICC) was used for mean two-rater participant scores, using a two-way mixed-effects model.³² Stability of participant scores across the two scenarios for each assessment was evaluated using the Wilcoxon signed-rank test. Effect size r was calculated to quantify between-group differences.³³ Analyses were conducted using Stata 16.0 (StataCorp, College Station, TX, USA), with significance level set at P<0.05. Data are presented as n (%), median (IQR), and odds ratio (95% confidence intervals). Missing data were excluded from analyses.

Results

Participants

Of 67 trainees invited to participate, 43 consented: 23 third-yr trainees (13 from Germany, 10 from Spain) and 20 fourth-yr trainees (11 from Germany, nine from Spain). Complete data were obtained from 85 encounters (46 from third-yr and 39 from fourth-yr trainees). Details of participant flow and data analysis are illustrated in Supplementary material 6, Supplementary Figure 1.

Participant characteristics

Third-yr trainees had a median age of 29.0 (27.5–31.5) yr; nine (39.1%) were male, two (8.7%) had >6 months of intensive care experience, and one (4.3%) was prehospital emergency physician. All third-yr trainees had prior exposure to simulation-based training, and 22 (95.7%) had participated in two or more simulation sessions. Fourth-yr trainees had a median age of 30.0 (28.75–31.0) yr; 16 (80.0%) were male, 17 (85.0%) had >6 months of intensive care experience, and five (25.0%) worked as prehospital emergency physicians. All had received simulation-based training on at least two occasions.

Evidence based on relationships with other variables

Fourth-yr trainees achieved significantly higher scores than third-yr trainees in the knowledge test, SACSS scores, and ANTS final scores, with medium effect sizes (Table 1). Significant positive linear correlations ranging from moderate to very strong³⁴ were observed among SACSS-checklist, SACSS-GRS, and ANTS scores in both scenarios: within-scenario (between-assessment tools), within-assessment (between scenarios), and cross-scenario (between tools). The knowledge test showed no correlation with the assessments in Scenario 1, and a moderate correlation with SACSS and ANTS scores in Scenario 2.³⁴ The strongest associations were between SACSS-GRS and ANTS within the same scenario (0.945 and 0.962 for Scenarios 1 and 2, respectively; Table 2).

Table 1.

Performance on the knowledge test, SACSS-checklist, SACSS-GRS, and ANTS assessments by year of training (third-yr vs fourth-yr trainees). SACSS-checklist, SACSS-GRS, and ANTS scores are presented as median and interquartile range (IQR). Final scores were calculated as the mean of the scores provided by the two evaluators. Values for 'Scenario 1 and 2' were calculated as the mean of the final scores obtained from both scenarios. ∗Performance data for one participant in Scenario 2 were not recorded and were excluded from the analysis. ^†Odds ratios and 95% confidence intervals were calculated using logistic regression. ^‡Two-tailed P-values were derived from Mann–Whitney U-test. ^¶Effect size r=|Z|/√(n₁+n₂). Effect size interpretation as proposed by Cohen: r<0.10: negligible; 0.10≤r<0.30: small; 0.30≤r<0.50: medium; r≥0.5: large.³³ ANTS, Anaesthetists’ Non-Technical Skills; GRS, global rating scale; SACCS, Structured Assessment of Clinical Skills in Simulation.

Assessment	Simulation scenario	Year of training		Odd ratio (95% confidence intervals)†	Mann–Whitney U-test	z	P-value‡	Effect size r¶
		Third yr (n1=23)	Fourth yr (n2=20)∗
Knowledge test	–	70.0 (64.0–73.0)	78.0 (69.0–82.0)	1.14 (1.03–1.26)	115.0	2.808	0.004	0.428
SACSS-checklist final score	Scenario 1	53.3 (48.9–60.0)	59.4 (56.7–63.9)	1.16 (1.03–1.31)	128.0	2.490	0.012	0.380
	Scenario 2	47.8 (43.3–58.9)	56.7 (54.4–61.1)	1.12 (1.03–1.23)	106.0	2.847	0.004	0.439
	Scenario 1 and 2	50.6 (45.6–60.0)	57.8 (56.1–61.7)	1.20 (1.06–1.35)	97.5	3.059	0.002	0.472
SACSS-GRS final score	Scenario 1	65.5 (56.0–83.5)	84.5 (70.5–91.5)	1.07 (1.02–1.13)	109.5	2.935	0.003	0.448
	Scenario 2	69.0 (55.5–86.0)	83.0 (77.5–89.5)	1.07 (1.01–1.13)	126.5	2.325	0.019	0.359
	Scenario 1 and 2	73.8 (54.5–84.2)	84.5 (76.0–90.2)	1.09 (1.02–1.15)	113.5	2.654	0.007	0.409
ANTS final score	Scenario 1	76.7 (62.9–88.3)	87.1 (79.2–91.5)	1.09 (1.02–1.16)	127.0	2.509	0.011	0.383
	Scenario 2	80.8 (64.6–87.9)	88.3 (82.1–93.8)	1.08 (1.01–1.15)	128.5	2.275	0.022	0.351
	Scenario 1 and 2	80.2 (60.8–86.9)	85.8 (80.4–91.2)	1.10 (1.02–1.19)	126.0	2.338	0.019	0.361

Table 2.

Pairwise linear correlation between scores obtained by participants on different assessments. Final scores were calculated as the mean of the scores provided by the two evaluators. Values represent Pearson correlation coefficients (PCCs) with corresponding P-values in brackets. Interpretation of PCC as proposed by Schober: PCC<0.10: negligible correlation; 0.10≤PCC<0.40: weak correlation; 0.40≤PCC<0.70: moderate correlation; 0.70≤PCC<0.90: strong correlation; PCC≥0.90: very strong correlation.³⁴ ∗Statistically significant at P<0.05. ^†Positive correlation between different assessment tools within the same scenario. ^‡Positive correlation between the same assessment tool across the two scenarios. ^¶Positive correlation between different assessment tools across different scenarios. ANTS, Anaesthetists’ Non-Technical Skills; GRS, global rating scale; SACCS, Structured Assessment of Clinical Skills in Simulation.

Variables	Knowledge test	SACSS-checklist final score for Scenario 1	SACSS-checklist final score for Scenario 2	SACSS-GRS final score for Scenario 1	SACSS-GRS final score for Scenario 2	ANTS final score for Scenario 1	ANTS final score for Scenario 2
Knowledge test	1.000
SACSS-checklist final score for Scenario 1	0.140 (0.369)	1.000
SACSS-checklist final score for Scenario 2	0.452∗ (0.003)	0.500∗‡ (0.001)	1.000
SACSS-GRS final score for Scenario 1	0.202 (0.193)	0.758∗† (0.000)	0.644∗¶ (0.000)	1.000
SACSS-GRS final score for Scenario 2	0.464∗ (0.002)	0.593∗¶ (0.000)	0.789∗† (0.000)	0.727∗‡ (0.000)	1.000
ANTS final score for Scenario 1	0.294 (0.056)	0.776∗† (0.000)	0.577∗¶ (0.000)	0.945∗† (0.000)	0.763∗¶ (0.000)	1.000
ANTS final score for Scenario 2	0.453∗ (0.003)	0.607∗¶ (0.000)	0.735∗† (0.000)	0.705∗¶ (0.000)	0.962∗† (0.000)	0.755∗‡ (0.000)	1.000

Evidence based on internal structure

For the SACSS-checklist in Scenario 1, both evaluators showed global agreement of >70% for 39 items and >80% for 31 of the 45 checklist items. The corresponding PABAK values were ≥0.41 for 39 items and ≥0.61 for 31 items, indicating predominantly moderate-to-substantial agreement according to the Landis and Koch classification.³⁵ In Scenario 2, evaluators achieved global agreement >70% for 37 items and >80% for 33 items, with PABAK ≥0.41 for 37 items and ≥0.61 for 33 items, again indicating moderate-to-substantial agreement.³⁵ The ICCs for the SACSS-checklist final score were 0.50 for Scenario 1 and 0.69 for Scenario 2, reflecting moderate reliability.³²

For the SACSS-GRS, ICC values for Scenario 1 categories ranged from 0.24 to 0.58, with a final score ICC of 0.67, indicating moderate reliability.³² For Scenario 2, category ICCs ranged from 0.33 to 0.72, with a final score ICC of 0.76, indicating good reliability.³²

For the ANTS scores, Scenario 1 ICC values ranged from 0.01 to 0.60, with a final score ICC of 0.52, resulting in a moderate reliability.³² Scenario 2 ICC values ranged from 0.14 to 0.68, with a final score ICC of 0.67, demonstrating moderate reliability.³²

ICC values for the final scores of each assessment tool are presented in Table 3. Agreement indices and ICC values for individual items within each instrument are detailed in Supplementary material 6, Supplementary Tables 3–8, and further discussed in Supplementary material 7.

Table 3.

Metrics of inter-rater reliability. SACSS-checklist, SACSS-GRS, and ANTS scores are presented as median and interquartile range (IQR). ∗Intraclass correlation coefficient (ICC) (95% confidence intervals) were calculated for ‘mean two-rater’ ratings using two-way mixed-effects model. ^†ICC interpretation as proposed by Koo and Li³²: ICC<0.50: poor reliability; 0.50≤ICC<0.75: moderate reliability; 0.75≤ICC<0.90: good reliability; ICC≥0.90: excellent reliability. ANTS, Anaesthetists’ Non-Technical Skills; GRS, global rating scale; SACCS, Structured Assessment of Clinical Skills in Simulation.

Assessment	Simulation scenario	Rater 1	Rater 2	ICC∗ (95% confidence intervals)	P-value	ICC interpretation†
SACSS-checklist final score	Scenario 1 (n=43)	62.2 (53.3–66.7)	53.3 (48.9–57.8)	0.50 (0.01–0.75)	0.000	Moderate
	Scenario 2 (n=42)	57.8 (48.9–62.2)	51.1 (44.4–57.8)	0.69 (0.28–0.86)	0.000	Moderate
SACSS-GRS final score	Scenario 1 (n=43)	80.0 (67.0–92.0)	76.0 (58.0–86.0)	0.67 (0.42–0.81)	0.000	Moderate
	Scenario 2 (n=42)	84.0 (57.0–89.0)	77.0 (68.0–88.0)	0.76 (0.59–0.86)	0.000	Good
ANTS final score	Scenario 1 (n=43)	90.0 (78.3–95.0)	78.3 (63.3–86.7)	0.52 (0.14–0.74)	0.000	Moderate
	Scenario 2 (n=42)	90.0 (75.0–93.3)	81.7 (71.7–91.7)	0.67 (0.46–0.81)	0.000	Moderate

In between-scenario comparisons for each instrument, SACSS-checklist scores were significantly lower in Scenario 2 than in Scenario 1, with medium effect size, whereas SACCS-GRS and ANTS scores showed no significant differences between scenarios (Supplementary material 6, Supplementary Table 9).

Post-simulation questionnaire

Results regarding scenario fidelity are presented in Supplementary material 6, Supplementary Tables 10 and 11. Participants generally reported a high level of agreement regarding the realism and fidelity of the simulation scenarios.

Discussion

Our expert panel designed simulation scenarios and assessment tools to assess ETR-defined competencies in anaesthesiology,³ similar to the approach used in the US Accreditation Council for Graduate Medical Education (ACGME) Milestones Competency Assessment project.¹⁶ This method ensures content validity, as it is based on current guideline recommendations, follows predefined assessment objectives, and aligns with expected trainee competencies outlined in the training curriculum, which are requisites for a robust simulation-based assessment.^9,14

Simulation-based assessment of clinical skills commonly uses checklists and GRS.^9,12 Several studies have used scenario-specific checklists,^17,36, 37, 38, 39, 40, 41, 42, 43, 44 whereas others have focused on GRS or a combination of both.^19,45, 46, 47 Both have proved validity, but checklists are scenario-specific, whereas GRS can be used across multiple scenarios.¹² We show significantly higher SACSS-checklist and SACSS-GRS scores among fourth-yr residents, which support discriminant validity. Similar approaches have been used to prove discriminant validity of technical skills assessment tools, either comparing trainees with anaesthesiology specialists,^38,41,42,44 or less experienced with more experienced residents.^{17,19,37,38,45} Similarly, the higher ANTS scores observed among fourth-yr trainees in our study support the discriminant validity of the tool. In line with our findings, an external validation study showed that the ANTS system effectively discriminated between second- and third-yr trainees.⁴⁸ The original validation by Fletcher and colleagues²⁸ established content validity through expert agreement on the tool's completeness, and high observability of the ANTS elements across ratings provided by 50 evaluators.

Our correlation analysis revealed no strong correlations³⁴ between knowledge test scores and SACSS (both checklist and GRS) or ANTS scores, consistent with previous findings. One study examined the correlation between anaesthesiology residents’ scores in technical and nontechnical skills during simulated scenarios and their official examinations results (United States Medical Licensing Examination Step 1 and American Board of Anesthesiology In-Training Examinations). No strong correlation was found, with coefficients ranging from r=–0.49 to r=0.49.³⁶ Similarly, another study found a low correlation (r=0.19) between simulation-based assessments of technical skills and written evaluations among medical students during their anaesthesiology rotation.⁴³ In another investigation, senior anaesthesia residents were evaluated on ‘resuscitation’ and ‘trauma’ using both simulation-based and oral examinations. A stronger within-scenario between-modality correlation was found (r=0.52 for ‘resuscitation’ and r=0.53 for ‘trauma’).⁴⁹ This lack of correlation could be partly explained by different tools measuring different competence domains.^36,43

In line with current validity frameworks,^11,30 a lack of correlation between instruments measuring different constructs constitutes validity evidence supporting the interpretation that each tool assesses a unique aspect of performance. Although knowledge tests assess core theoretical knowledge, simulation-based assessment require practical knowledge implementation and hands-on skills. Furthermore, oral examinations demand a deeper level of understanding than written or multiple-choice tests, which could account for the stronger correlations observed in one study.⁴⁹ Previous studies have examined the correlation between checklists and GRS for assessment of clinical skills during simulation-based evaluations. High correlations have been reported in medical students (r=0.74),⁵⁰ during anaesthesia board examinations (r=0.77),²⁰ and a broader range of health professions (r=0.76).¹² Consistent with these findings, our results also showed a strong correlation³⁴ between the SACSS-checklist and the SACSS-GRS for each scenario. As both instruments were designed to assess the same construct (clinical skills), this correlation provides further validity evidence in support of the assessments.^11,30 Correlations were weaker for SACSS-checklist scores, both between scenarios and with other assessments across scenarios, consistent with the checklist's high specificity. Conversely, strong between-scenario correlation³⁴ for SACSS-GRS and ANTS scores were found, suggesting that high performers scored better throughout the scenarios, which is a further argument for the validity of simulation-based assessments. Interestingly, we found a very strong within-scenario between-assessment correlation³⁴ between SACSS-GRS and ANTS (i.e. SACSS-GRS and ANTS for the same scenario), with a strong³⁴ between-scenario between-assessment correlation (i.e. SACSS-GRS for Scenario 1 and ANTS for Scenario 2 and vice versa). This might reflect that more proficient anaesthesiologists tend to perform well in both technical and nontechnical domains. Moreover, holistic GRS such as the SACSS-GRS might partially capture nontechnical competencies, embedded within what has been described as 'nuanced elements of expertise'.¹²

Our findings indicate moderate inter-rater reliability for the SACSS-checklist and moderate-to-good inter-rater reliability for the SACSS-GRS, with results within the range prior research. The inter-rater reliability of SACSS-checklist (ICC 0.50 to 0.69) was comparable with previously reported values (ICC 0.50 to 0.85),¹⁷ but lower than those found in other studies (ICC 0.70 to 0.95; ICC 0.84 to 0.95).^43,51 For the SACSS-GRS, the inter-rater reliability (ICC 0.67 to 0.76) exceeded that reported in one study (ICC 0.45),⁵² but was lower than that found in others (ICC 0.77 to 0.87; ICC 0.70 to 0.87).^49,51 Other studies investigating the inter-rater reliability of checklists and GRS have reported a wide range of results, using methodological approaches that differ from those used in our study.^{20,37,38,40,41,44,47,53,54} As such, direct comparisons should be interpreted with caution.

The inter-rater reliability of the ANTS tool was moderate, with values falling within the range reported in previous studies. The range of ICC values obtained for individual ANTS elements (0.01 to 0.68) was similar to those reported in earlier work (ICC 0.11 to 0.62),⁵⁵ but lower than those found in a more recent study (ICC 0.33 to 0.78).⁴⁸ The ICC values for ANTS final scores in both scenarios (0.52 to 0.67) were lower than those reported in some investigations (ICC ≥0.97; ICC 0.97; ICC 0.88),^36,56,57 although comparable with that found in another study (ICC 0.74).⁵⁸

Finally, SACSS-checklist scores differed significantly between scenarios, whereas SACCS-GRS and ANTS scores were stable. This could reflect differences in scenario difficulty reported elsewhere.^{17,20,46,47,59,60} However, because the SACSS-checklists were bespoke to each scenario, whereas SACSS-GRS and ANTS were applicable to both, we interpret the isolated checklist difference as indicative of greater checklist stringency, consistent with previous findings.⁶¹

The newly developed SACSS and the established ANTS tools provided validity evidence based on relationships with other variables, demonstrating both convergent and discriminant patterns, and internal structure, reflected by moderate-to-good inter-rater reliability and cross-scenario consistency. These findings support the use of both instruments to assess ETR-defined competencies among advanced anaesthesiology trainees in Germany and Spain. To our knowledge, this is the first study to collect validity evidence for a simulation-based assessment of both technical and nontechnical competencies across two countries. Similar to a previous study from Canada,¹⁹ we successfully implemented the simulation-based assessment in two languages, reinforcing the feasibility of conducting multilingual, multicentre simulation-based assessments within the European context. Additionally, the scenarios and associated assessment tools were specifically designed to address two of the top 10 critical clinical situations for which all anaesthesiology trainees should be trained, according to a recently published expert consensus on simulation in specialist anaesthesiology training.⁶² Collectively, our findings represent a valuable contribution to the development of simulation-based training and assessment within the European postgraduate anaesthesiology curriculum, where simulation plays an increasingly important role.

Several limitations to this study should be considered. Trainees were categorised according to their officially assigned year of training at the time of participation. As the total number of months of training was not collected, variation in experience both between and within years (e.g. trainees at the beginning vs the end of the same year) might have introduced variability in competence levels among participants. Voluntary participation might have introduced selection bias, favouring trainees with a particular interest in simulation. Nevertheless, although intrinsic motivation is known to influence academic performance, prior research found no significant correlation between motivation and nontechnical skills performance in simulated scenarios.⁶³ Although a multicentre study using different manikins and simulation teams showed good inter-simulator reliability,⁴⁰ variability between simulators and human factors remain sources of potential bias. Furthermore, using only two scenarios limits the breadth of assessed competencies. Although studies investigating validity of simulation-based assessments use between one and eight scenarios,^17,19,37,39, 40, 41, 42, 43, 44, 45, 46, 47, 48 a study investigating the minimal number of scenarios and raters required for a reliable simulation-based assessment concluded that four scenarios per participant evaluated by two experts were necessary.⁶⁴

Despite the consistency of our findings, the absence of a formal assessment of data saturation means that the inclusion of additional scenarios or participants might have yielded different results, thus limiting the generalisability of our study. Finally, there are some limitations regarding the assessment tools and the scoring process. The primary limitation of the SACSS-checklist is its scenario specificity: it was designed to assess performance in a particular situation and is not generalisable to other scenarios. In line with previous studies,^{17,37,38,41,43,47,53,54} we used a binary scoring system. Other studies have instead used ternary scoring schemes (e.g. ‘not performed’, 0; ‘performed incompletely’, 1; ‘performed correctly’, 2)^44,51 or more complex weighted scales based on the timeliness of critical actions.⁴⁰ Although our approach was intended to simplify the use of the tool, a limitation raised by the raters was the lack of an intermediate scoring category (e.g. 'partially performed/performed with assistance'), which made it difficult to capture the full range of performance. The SACSS-GRS used a consistent five-point Likert scale for both individual domains and overall performance. This could have introduced cognitive bias, with raters potentially aligning the overall score with the average of domain-specific ratings. Furthermore, the absence of formal rater calibration and the fact that one of the SACSS creators also acted as a rater might have contributed to discrepancies in scoring between raters, partially explaining the moderate reliability observed with the SACSS.

With regard to the ANTS, one constraint of the study was raters' unfamiliarity with the system and the limited training they received, in contrast to other studies.^28,55,56 Unlike some studies, we followed the original methodology.²⁸ Previous research has demonstrated that rater training enhances scoring accuracy,^56,57 and it is likely that our raters' competence improved over time. To mitigate bias, all performances were assessed in a randomly assigned order; however, this might have inadvertently reduced reliability. These factors could account for the lower reliability results observed in our study compared with those reported elsewhere.^36,43,48,49,56, 57, 58 Our scoring of technical and nontechnical skills was conducted through video review. Although acceptable reliability has been reported between live scoring and delayed video review,¹⁶ we believe that the opportunity to review recordings multiple times might influence the scoring process and limit the generalisability of our findings to other settings. Finally, as simulated scenarios do not perfectly replicate real clinical situations, participant performance can differ from that in actual healthcare settings, thereby limiting the generalisability of any simulation-based assessment and its use for high-stakes examinations.

In line with the European consensus on competency-based postgraduate training in anaesthesiology,² simulation-based assessment should be seen as one element within a broader, multimodal strategy, and combined with complementary methods to ensure a comprehensive evaluation of trainee competence. Future research should focus on the development and validation of additional scenario-specific assessment instruments, aligned with national and European guidelines, to support evaluation of clinical skills across a range of simulated medical conditions, clinical environments, and cultural contexts. Comparative studies of existing assessment tools are warranted to identify those that demonstrate the greatest validity, reliability, and usability for integration into practice. Research involving broader cohorts of European trainees and diverse healthcare systems is also needed to gather further validity evidence for simulation-based assessments, and to investigate their feasibility and applicability, both formative and summative, within the wider context of European postgraduate training. Establishment of standardised pass–fail criteria for high-stakes, summative simulation-based assessments should be the focus of rigorously designed studies. Advancing the field requires collaboration among medical education and simulation researchers, national stakeholders, and European organisations such as the EBA and ESAIC.

In conclusion, this study supports the validity and reliability of simulation-based assessments for evaluating ETR-defined competencies in advanced anaesthesiology trainees across European centres.

Authors' contributions

Conceived and designed the study: GSAF, LGL, RV, CN

Data acquisition: LGL, BT, MLB, JMP, CI, RB, TC, JS, AC, MFP

Data analysis and interpretation: GSAF, SGB, FM, BT, MV

Statistical analyses: FM

Drafted the manuscript: GSAF, BT, MV, FM, RV, CN

Critically revised the manuscript and approved the final version to be published: all authors

Agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved: all authors

Funding

The Department of Anaesthesiology, Heidelberg University (Germany); the Department of Anaesthesiology and Critical Care, Hospital Clínic de Barcelona (Spain); and the Skills Laboratory, Facultat de Medicina i Ciències de la Salut, Universitat de Barcelona (Spain).

Declaration of generative AI and AI-assisted technologies in the writing process

During the preparation of this work, the authors used ChatGPT in order to improve language and readability. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication.

Declaration of interest

The authors declare that they have no conflict of interest.

Handling Editor: Hugh C Hemmings Jr

Appendix A. Supplementary data

The following are the Supplementary data to this article: