Competency based medical education: assessment of procedural and cognitive skills in gastroenterology residency training

Abstract

Introduction

Competence by Design (CBD) was implemented in Canada to assess physician trainees’ ability to demonstrate competence via entrustable professional activities (EPAs) utilizing the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE); a score originally validated to assess surgical competence. Our study assessed for differences in O-SCORE utilization between non-procedural (cognitive) and procedural EPA assessments in a real-world setting to inform future critical appraisal of CBD.

Methods

Adult Gastroenterology subspecialty EPA assessments from 2019 to 2023 at the University of Alberta were reviewed. Evaluator gender, clinical vs. academic practice, advanced training expertise, EPA type, and O-SCOREs were extracted from each EPA assessment. Competence assessments (achieved, neutral, not achieved) were assigned to each EPA assessment based on local competence committee protocols. Chi-squared testing with 95% confidence intervals were calculated.

Results

2660 EPA assessments were included (1385 cognitive and 1275 procedural). Approximately 70% of total EPA assessments denote competence, with approximately 20% neutral, and < 10% indicating “competence not yet achieved”. Proportionally higher cognitive EPA assessments denoted competence compared to procedural EPA assessments (75% vs. 63%, p < 0.01). Evaluator characteristics associated with EPA achievement were male gender, community (vs. academic), and hepatology (vs. luminal GI) practice.

Discussion

We report low rates of the full range of O-SCORE utilization and the majority amount of EPA assessments denoting competence. Further the proportion of EPAs assessed as success varied by evaluator characteristics (gender, specialty, and practice type). This stresses the need for continued cycles of critical appraisal to improve faculty assessment and feedback in residency trainee progress.

Introduction

Competence by Design (CBD) is the approach of competency-based medical education for specialty residents in Canada as of 2017 This system was created by the Royal College of Physicians and Surgeons of Canada (RCPSC) and has as its foundation the very definition of competence [1]. Competency is typically defined as the ability to perform a task efficiently and successfully; however, within the domain of medical education, it is considered to be a specific stage of training in the progression towards medical expertise [2]. A working definition in the realm of medical education has been posited by Carraccio et al. as: “a complex set of behaviors built on the components of knowledge, skills, attitudes, and ‘competence’ as personal ability” [3]. The pathway to competence has been an evolving framework with origins as far back as 60 years [4]. More modern iterations for a competence based training model have been adapted from a 5-step framework progressing through the stages of novice, advanced beginner, competence, proficiency, and eventually expertise [5, 6]. This linear pathway subsequently evolved into a more logarithmic growth curve through these 5 stages via deliberate clinical practice [2].

CBD falls under the umbrella of Competency Based Medical Education (CBME) and is a divergence from the previously labeled “process and structure” based approach to medical education derived from the original Flexner Report [6, 7]. CBME shifts the focus from process-related goals to those of “learner and program related” outcomes [4].

These outcomes expected of a medical professional are then amalgamated into a competency framework. In Canada this framework is known as the CanMEDS Roles [8].

CBD assesses the ability of physician trainees to demonstrate competence in the CanMEDS framework via entrustable professional activities (EPAs). EPAs were designed as the bridge from theory to action in the implementation of CMBE [9]. EPA assessments utilize the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) as the rubric for evaluating competence [10]. This one to five scoring system is a follows: (1) “I had to do”; (2) “I had to talk them through”; (3) “I had to prompt them from time to time”; (4) “I needed to be in the room just in case”; and (5) “I did not need to be there” [10]. The O-SCORE was originally created for and validated in different surgical settings, yielding reproducible distinguishability between senior and junior learners [10, 11].

Importantly, there remains a paucity of data validating the O-SCORE in non-procedural EPA evaluations, what we have termed cognitive EPAs. Cognitive EPAs similarly evaluate the competencies outlined in the CanMEDS Roles, but focus on skills outside of a procedural setting [8]. For example, a cognitive EPA may assess a trainee’s ability to demonstrate the CanMEDS medical expertise role by taking a patient’s medical history, rather than a procedural demonstration of medical expertise by performing a colonoscopy. The O-SCORE has been implemented to assess all skills across medical and surgical specialties under the RCPSC. This type of assessment is grounded in the principle of observation, both direct (direct observation of the encounter by the evaluator) and indirect (review of the clinical encounter or documentation by the evaluator after the fact).

Adult gastroenterology subspecialty residency training programs adopted CBD in 2019 and require a mix of procedural (endoscopy) and cognitive EPAs for successful promotion and completion of training. Adult gastroenterology residency in Canada requires a demonstration of competence for 16 cognitive and 5 procedural EPAs. Each EPA requires the assessment of multiple different clinical scenarios; thus, requiring the demonstration of competence in 82 cognitive and 57 procedural EPA evaluations. Endoscopic procedures are directly observed (as faculty are physically present during procedures), while the cognitive type EPA assessments are more frequently indirectly observed. Examples of indirect observations in a cognitive EPA assessment may include the review of a completed consultation or review of documentation rather than a direct observation of the patient encounter.

Moreover, assessor expertise is recognized as playing a crucial role in performance assessment. Previous literature has identified three such categories that affect evaluations: assessor characteristics, assessor perceptions of the task, and the context of the assessment [12]. The Division of Gastroenterology at the University of Alberta has a large and diverse cohort of staff physicians based on gender, academic/clinical rank, and clinical areas of expertise. This unique combination affords the opportunity to investigate the real-world impact of both evaluator characteristics and O-SCORE usage for cognitive assessments in CBD.

Our study aims to assess differences in O-SCORE utilization between procedural and cognitive EPA assessments, as well as the impact of assessor characteristics on O-SCORE usage in order to provide insight into the challenges faced in the implementation of CBD and inform future directions for competency-based assessment.

Materials and methods

Study population and data collection

We performed a single-center, retrospective observational study of all EPA assessments completed in the evaluation of adult gastroenterology subspecialty residents training at the University of Alberta during a study period of July 2019 to January 2023. During the study period 14 residents were evaluated with training program expectations to complete an average of 3 EPA attempts per week. All trainees included in the present study were able to be assessed by all included evaluators. 54 evaluators were present at our center during the study period.

Every evaluator included within this study has received formal training on CBD/EPAs for the assessment of resident trainees.

EPA data was obtained from a pre-existing Post-Graduate Medical Education database at the University of Alberta. The raw data were anonymized through the generation of a unique randomized code for each evaluator by the data custodian prior to granting access to the remainder of the study team to avoid possible identification of individual evaluators. All written feedback outside of the O-SCORE rating was excluded to prevent the identification of individual trainees. For EPA assessments to be included, the evaluating staff physician was required to be actively practicing medicine within the Faculty of Medicine and Dentistry, Division of Gastroenterology at the University of Alberta during the specified study period. Evaluators completing less than five total EPA assessments were excluded as they would not have the ability to utilize the entirety of the 5-point O-SCORE scale.

Data extraction

Extracted variables from each EPA assessment included: the EPA identifier outlined by the RCPSC and the O-SCORE given for the assessment. Based on each EPA identifier, an EPA type was assigned (cognitive vs. procedural). The RCPSC provides guidance on which adult gastroenterology EPAs are considered procedural. All EPAs not indicated as procedural by the RCPSC were categorized as cognitive EPAs. Locally, a score of 5 denotes competence, while a 1–3 indicates competence was not yet achieved. A score of 4 may be accepted as evidence of competence at the discretion of the local competency committee and thereby has been termed “neutral” for this study.

Each EPA assessment completed also had evaluator characteristics extracted including: gender, academic vs. clinical practice, and area of clinical expertise within gastroenterology (e.g. inflammatory bowel disease, hepatology, therapeutic endoscopy, general gastroenterology). Motility and nutrition experts were classified as general gastroenterology to avoid identification of an individual evaluator, as there were only two clinical experts in each area at our site. Clinical expertise was then categorized as hepatology vs. luminal gastroenterology (all other areas of clinical expertise) for analysis.

Data analysis

Analysis was conducted using IBM SPSS Statistics Version 29.0 (IBM Corporation, Armonk, NY). The absolute number and proportion of total, procedural, and cognitive EPA assessments were calculated for each value of the O-SCORE as well as for competence designations (competence achieved, neutral, and competence not achieved). Chi-square testing was completed to assess for differences in observer characteristics (gender, academic vs. clinical practice, area of clinical expertise within gastroenterology) for each value of the O-SCORE and for competence designations. Spearman’s rank correlation coefficients were calculated to assess for monotonic correlation between the number of EPA evaluations completed and the utilization of the O-SCORE. 95% confidence intervals were calculated under the assumption of a Poisson distribution. A p-value of < 0.05 was regarded as significant.

Results

Evaluator demographics

Forty-seven evaluators met our inclusion criteria with 7 faculty excluded due to an insufficient number of submitted EPA evaluations. The full demographic data of the evaluators is shown in Table 1. When grouped for analysis, there were 34 luminal gastroenterologists and 13 hepatologists.

Table 1.

Demographics data of all evaluators including number of Entrustable professional activities (EPA) completed by each demographic category stratified by EPA type

	Number of Staff (n, %)	Total EPAs Completed (n, %)	Cognitive EPAs Completed (n, %)	Procedural EPAs Completed (n, %)
Total Staff:	47 (100)	2660 (100)	1385 (52.1)	1275 (47.9)
Gender:
Male	30 (63.8)	1774 (66.7)	937 (67.7)	837 (65.6)
Female	17 (36.2)	886 (33.3)	448 (32.3)	438 (34.4)
Practice Type:
Academic	24 (51.1)	1488 (55.9)	912 (65.8)	576 (45.2)
Clinical	23 (48.9)	1172 (44.1)	473 (34.2)	699 (54.8)
Clinical Expertise:
Luminal	34 (72.3)	1428 (53.7)	410 (29.6)	1018 (79.8)
Hepatology	13 (27.7)	1232 (46.3)	975 (70.4)	257 (20.2)

EPa characteristics

A total of 2660 EPA assessments were included, comprised of 1385 cognitive and 1275 procedural EPA assessments as outlined in Table 1. The number of EPA assessments completed by each evaluator ranged from 11 to 157, with a mean of 55 (SD 39.8) and median of 43 (IQR 25.8, 68.8). The number of cognitive EPA assessments completed by each evaluator ranged from 5 to 87 with a mean of 30 (SD 23.9) and median of 25 (IQR 10.3, 50.5).

The absolute number of procedural EPA assessments completed by each evaluator ranged from 6 to 79 with a mean of 27 (SD 20.3) and median of 24 (IQR 13.5, 34.5).

Approximately 70% of total EPA evaluations were assessed as “competence achieved”, with approximately 20% evaluated as neutral, and less than 10% of all EPA assessments indicated “competence was not yet achieved”.

When separated by EPA type, procedural EPA assessments had a lower proportion indicating competence was achieved compared to cognitive EPAs (63% vs. 75%, p < 0.01). In keeping with this pattern, procedural EPAs had a slightly higher proportion of evaluations submitted as “neutral” (26.0% vs. 20.3%, p < 0.01) or “competence not achieved” (10.4% vs. 4.5%, p < 0.01) compared to cognitive EPAs. When stratified by the O-SCORE, zero cognitive EPA assessments were completed with a score of 1/5, and < 1% had a rating of 2/5. There were proportionally more procedural EPAs evaluated at 1/5, 3/5, and 4/5 (p < 0.01) when compared to cognitive EPAs. Full details of EPAs stratified by competence evaluation and O-SCORE rating are outlined in details in Fig. 1, and Table 2.

Fig. 1.

Proportion of Entrustable Professional Activities (EPAs) stratified by EPA type compared to competence assessments

Table 2.

Chi squared testing of Entrustable Professional Activities (EPA) competence assessments and O-SCORE’s vs. EPA type

	Cognitive	Procedural	P-value
Competence:
Not Achieved	4.5% (63/1385)	10.4% (132/1275)	< 0.01
Neutral	20.3% (281/1385)	26.0% (332/1275)	< 0.01
Achieved	75.2% (1041/1385)	63.6% (811/1275)	< 0.01
O-SCORE:
1/5	0.0% (0/1385)	1.7% (22/1275)	< 0.01
2/5	0.4% (5/1385)	0.9% (12/1275)	0.06
3/5	4.2% (58/1385)	7.7% (98/1275)	< 0.01
4/5	20.3% (281/1385)	26.0% (332/1275)	< 0.01
5/5	75.2% (1041/1385)	63.6% (811/1275)	< 0.01

O-SCORE utilization

Most commonly, evaluators only utilized 3 of the possible 5 options on the O-SCORE for all EPA evaluations. Figure 2 demonstrates a trend toward higher O-SCORE utilization as the median number of EPA evaluations completed increases, but also highlights broader interquartile ranges. There was a weak correlation between O-SCORE utilization and the number of EPA evaluations completed by an evaluator (r_s = 0.33, p < 0.5). In this data, one evaluator completed 141 EPA evaluations utilizing only 2 of 5 O-SCORE options, one evaluator completed 46 EPA evaluations utilizing only a score of 5/5 for all assessments, and one evaluator utilized all 5 options having completed as few as 15 EPA evaluations. Every evaluator that utilized just 2 of the 5 possible options on the scale submitted only scores of 4/5 or 5/5 on the O-SCORE. When stratified by EPA type, evaluators most commonly utilized 2 options for cognitive EPAs and 4 options for procedural EPAs of the possible 5 options on the O-SCORE. No clear trend in median number of EPA evaluations completed was seen in relation to O-SCORE utilization for procedural EPA evaluations [data not shown]. There was no significant correlation between O-SCORE utilization and the number of EPA evaluations completed by an evaluator for procedural EPA evaluations (r_s = 0.24, p = 0.11). Similar to overall EPAs, a higher median number of cognitive EPA evaluations completed by an individual evaluator showed a trend toward higher O-SCORE utilization (up to and including utilization of 4 out of 5 options) with moderate correlation (r_s = 0.51; p < 0.5). However, this trend is interrupted as zero evaluators utilized all 5 options on the O-SCORE for cognitive EPA assessments. For procedural EPA assessments, three evaluators utilized only 1 option on the O-SCORE and thirteen evaluators utilized 2 options, with a range of 10–33 and 6–79 procedural assessments completed per evaluator respectively. For cognitive EPA assessments, six evaluators utilized only 1 option on the O-SCORE and seventeen evaluators utilized 2 options, with a range of 10–63 and 6–65 cognitive assessments completed per evaluator respectively.

Fig. 2.

Box and whisker plot of the number of different options utilized on the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) out of 5 compared to the number of Entrustable Professional Activities (EPAs) completed by each evaluator

Evaluator characteristics

When separated by evaluator gender, female evaluators submitted a slightly greater proportion of all EPA assessments scored as neutral (p = 0.04) or not achieved (p < 0.01) compared to male evaluators; this trend was also true for cognitive EPAs but there were no statistically significant differences between male and female evaluators for the proportion of procedural EPA assessments submitted as competence achieved (p = 0.17) or neutral (p = 0.57).

All statistically significant differences when separated by EPA type are minimal, with full data outlined in Table 3A.

Table 3.

Chi squared testing of entrustable professional activity competence assessments stratified by (A) evaluator gender, (B) evaluator practice type, and (C) evaluator clinical expertise

	Male	Female	P-value
All:
Not Achieved	6.0% (106/1774)	10.0% (89/886)	< 0.01
Neutral	21.7% (385/1774)	25.7% (228/886)	0.04
Achieved	72.3% (1283/1774)	64.2% (569/886)	0.02
Cognitive:
Not Achieved	3.5% (33/937)	6.7% (30/448)	< 0.01
Neutral	18.4% (172/937)	24.3% (109/448)	0.02
Achieved	78.1% (732/937)	69.0% (309/448)	0.07
Procedural:
Not Achieved	8.7% (73/837)	13.5% (59/438)	0.01
Neutral	39.7% (332/837)	27.2% (119/438)	0.57
Achieved	65.8% (551/837)	59.4% (260/438)	0.17

	Academic	Clinical	P-value
All:
Not Achieved	9.9% (147/1488)	4.1% (48/1172)	<0.01
Neutral	26.6% (396/1488)	18.5% (217/1172)	<0.01
Achieved	63.5% (945/1488)	77.4% (907/1172)	<0.01
Cognitive:
Not Achieved	6.5% (59/912)	0.8% (4/437)	<0.01
Neutral	22.6% (206/912)	15.9% (75/473)	0.01
Achieved	70.9% (647/912)	83.3% (394/473)	0.01
Procedural:
Not Achieved	15.3% (88/576)	6.3% (44/699)	<0.01
Neutral	33.0% (190/576)	20.3% (142/699)	<0.01
Achieved	51.7% (298/576)	73.4% (513/699)	<0.01

	Hepatology	Luminal	P-value
All:
Not Achieved	2.7% (18/667)	8.9% (177/1993)	<0.01
Neutral	13.5% (90/667)	26.2% (523/1993)	<0.01
Achieved	83.8% (559/667)	64.9% (1293/1993)	<0.01
Cognitive:
Not Achieved	2.4% (10/410)	5.4% (53/975)	0.02
Neutral	12.0% (49/410)	23.8% (232/975)	<0.01
Achieved	85.6% (351/410)	75.2% (690/975)	<0.01
Procedural:
Not Achieved	3.1% (8/257)	12.2% (124/1018)	<0.01
Neutral	16.0% (41/257)	28.6% (291/1018)	<0.01
Achieved	80.9% (208/257)	59.2% (603/1018)	<0.01

Evaluators with clinical medical practices submitted proportionally more EPA assessments denoting competence when compared to academic staff (77.4% vs. 63.5%, p < 0.01). This pattern was evident in both cognitive and procedural EPA subgroups. The most pronounced difference was evident for procedural EPAs (51.7% vs. 73.4%, p < 0.01). Full details are outlined in Table 3B. When stratified by O-SCORE’s, clinical evaluators gave proportionally more 5/5 ratings (p < 0.01), but lower 3/5 and 4/5 ratings compared to academic staff (p < 0.01, < 0.01 respectively) [data not shown].

Hepatologists submitted proportionally more evaluations indicating competence was achieved when compared to luminal gastroenterologists (83.8% vs. 64.9%, p < 0.01). This pattern was consistent for both cognitive and procedural EPA subgroups. Full details are outlined in Table 3C. When stratified by the O-SCORE, clinical hepatologists gave proportionally more 5/5 ratings, but lower 3/5 and 4/5 ratings compared to luminal gastroenterologists (p < 0.01 for all comparisons) [data not shown].

Discussion

The implementation of CBME in Canadian Medical Education has been a fundamental shift in educational theory that has been accompanied by numerous innovations including the creation of competence committees, generation of electronic trainee evaluation portfolios, time variable credentialing, and a system that allows real-time low-stakes assessments [13]. As with any large-scale structural change in medical education, challenges have been encountered by both evaluators and trainees with potential impacts on the true representation of a trainee’s progress toward competence [13, 14]. We aimed to provide a real-world evaluation of CBD by assessing O-SCORE utilization stratified by EPA type (procedural vs. cognitive) and the effects of evaluator characteristics on competence evaluations to highlight the importance of cyclical critical appraisal within CBME and ensure resident physicians are receiving high-quality education to cultivate a competent workforce of future physicians.

We report that a large majority of all EPA assessments indicate that competence was achieved (approximately 70%) with a significant difference in proportion when comparing cognitive vs. procedural EPAs (75% vs. 63% respectively, p < 0.05). In line with this pattern, we demonstrate low overall utilization of the full range of the O-SCORE. We report a weak correlation between a greater utilization of the O-SCORE by evaluators who have completed more total EPA evaluations, which may signal the role sample size plays in the opportunity to utilize the entire scoring spectrum or perhaps increased experience leading to improved evaluation skills and a deeper understanding of how to utilize the whole scale appropriately. Further investigation of this correlation in larger scale prospective studies may help elucidate valuable information regarding the role evaluator experience plays in this context. Other factors beyond the number of EPAs completed likely influence this data set. One example is related to the significant outliers for individual evaluator’s usage of the O-SCORE, such as the evaluator completing 141 EPA evaluations utilizing only 2 of 5 O-SCORE options. Strikingly, zero EPA assessments were submitted as a 1/5 O-SCORE rating across 1385 cognitive EPA evaluations, translating to zero staff having utilized all 5 options of the O-SCORE for cognitive EPA assessments. When stratified by EPA type, evaluators most commonly utilized more options on the O-SCORE for procedural vs. cognitive EPA assessments. Again, a low number of EPAs completed by an individual assessor may play a role in data interpretation, however assessors utilizing only one option on the O-SCORE ranged up to a maximum of 33 and 63 completed procedural and cognitive EPAs, respectively. In our subset analysis, we highlight differences in correlation between O-SCORE utilization and the number of EPA evaluations completed by an assessor. Interestingly, we found no correlation between these variables for the validated context of procedural EPAs, but did find correlation for cognitive EPAs up to 4/5 utilized options. This difference may indicate that assessors are unclear or resistant to the use of the entire O-SCORE spectrum in cognitive assessments. Our data highlights an asymmetric distribution in this real-world implementation of CBD favoring assessments of “competence achieved” for both EPA types (though more pronounced in cognitive EPA assessments), and highlights opportunities for further research to determine if such patterns are influenced by bias or represent an accurate benchmark of evaluations in CBME.

One potential consideration that may influence our results relates to the O-SCORE being co-opted for use in cognitive EPA evaluations without formal validation in this context [10, 11, 15]. Previously, O-SCORE utilization was assessed in multiple controlled training settings [16–18]. A modified version of the O-SCORE has been shown to correlate with training level in an objective structured clinical examination (OSCE), but did not show significant correlation during communication-based evaluation [16]. A modified evaluation modality termed the Ottawa Clinical Assessment Tool (OCAT) showed reliable correlation to a trainee’s residency level in a clinical setting, however, one OCAT assessment contains 9–11 discrete O-SCORE ratings within one evaluation that extends longitudinally across multiple clinical encounters, unlike the discrete low-stakes context of EPA assessments within CBD [17]. These examples provide promise for the successful use of the O-SCORE in cognitive assessments, however direct comparison to our present data evaluating the current iteration of EPA assessments in CBD is limited due to the modified scoring tools used in those studies.

Cognitive EPAs assessed in a simulation training setting have been shown to lack correlation to real-world EPA assessments [18]. This lack of correlation highlights complexity for the study of O-SCORE utilization for cognitive assessments, as intricate social and interpersonal interactions likely compound sources of bias when taken out of highly controlled simulation settings and limit the ability to methodologically study the validity of the O-SCORE in this context. From a trainee perspective, there has been optimism about this new system of evaluation; however, apprehensions were voiced regarding the effort and time required for EPA evaluations and the potential for a poor culture of “buy-in” from both trainees and assessors leading to heterogeneity in quality feedback [19–23]. The increased evaluation burden affecting trainee wellness is thought to predispose to “gaming of the system” through preferential evaluator selection and requesting evaluations only when competence was felt to be demonstrated, ultimately inhibiting the growth mindset and focus on CanMEDS roles that CBME was designed to foster [13, 14, 21–24].

Additionally, evaluators have perceived that the language of the O-SCORE does not always authentically reflect their true decision regarding competence in a non-procedural setting [25]. These sources of previously identified bias lend context to our data, showing the overwhelming proportion of EPA observations are rated 4/5 and 5/5 for both procedural and cognitive EPAs. Our data suggest that these trends are multifactorial rather than solely a byproduct of O-SCORE utilization for cognitive assessments. Increasingly, these concerns over bias and validation fuel the call for ongoing critical appraisal to ensure that EPA assessments accurately reflect real-world trainee competence and patient-related outcomes, with suggestions that this may ultimately require evaluation beyond the O-SCORE used in current EPA assessments [26].

With underutilization of the lower scores (1–2/5) of the O-SCORE in cognitive EPA evaluations, a 3-point scoring system may be better suited for cognitive-type task assessment. We suggest a 3-point modified O-SCORE as follows: (1) Competence not achieved, (2) Competence nearly achieved with room for improvement, and (3) Competence achieved. This modified scale may help to alleviate the perceived ambiguity of language for the assessment of competence in cognitive EPAs, and lead to a more complete utilization of the scale when compared to the current 5-point O-SCORE [25]. This 3-point scale becomes particularly pragmatic in the assessment of more senior medical trainees when competence is more likely to be demonstrated, but there remain areas for improvement toward expertise [5, 6]. Validation of such a scoring system as well as direct comparison to the currently established utilization of the O-SCORE for cognitive EPA assessments would lay the critical groundwork for the navigation of future research in this area and may allow more linear assessment of real-world biases affecting the high quality implementation of CBD in Canada.

Our findings support previous literature which has indicated that assessor characteristics impact evaluations [12, 24]. Evaluator gender in our study showed little effect on evaluations with female evaluators submitting only a mildly increased proportion of EPA assessments categorized as neutral (p = 0.04) and not achieved (p < 0.01).

Interestingly, evaluators with clinical practices or clinical expertise in hepatology submitted proportionally more EPA assessments indicating competence was achieved across total, cognitive, and procedural assessments. The reason for this trend is not directly assessed in the present study, though our data may possibly be explained by the previously described “hawk and dove effect” wherein evaluators who are perceived to be more lenient assessors were in fact more likely to submit above-average assessments of medical trainees [24]. It is possible that there are more “doves” present within our local hepatology and clinical practice evaluators. We identified a numerical trend demonstrating evaluators with professorship status, across all areas of advanced training expertise, completed more EPA assessments denoting competence [data not shown] which may help account for our findings. The biases inherent to this study limit the ability for causal relationships of evaluator characteristics to be assessed, but lay the foundation for future studies into this topic.

The strengths of our study stem from the unique mix of EPA types required in an adult gastroenterology subspecialty residency program, wherein a large volume of both cognitive and highly technical procedural EPAs are required to be deemed a competent gastroenterologist. Limitations of our study include the sources of bias inherent to a retrospective observational study. Ethics approval was not granted to assess trainee characteristics due to concerns of easy trainee identification. As such, we were unable to assess the role trainee characteristics may play in the real-world implementation of CBD in the present study. This limitation inhibits our ability to assess possible sources of bias such as the introduction of nesting observations for each trainee. Fortunately, the framework of CBD does have inherent structural characteristics which limit the degree of nesting within evaluations. CBD has 4 stages that a trainee must progress through to complete their residency training successfully [1]. Those stages are: Transition to Discipline, Foundations of Discipline, Core of Discipline, and Transition to Practice [1]. Once a trainee demonstrates competence in the required EPAs of each stage, they are promoted to the next stage and no longer complete assessments from the earlier stages; thus, limiting potential nesting effects of repeating the same EPA assessment multiple times. This framework of CBD does not account for the potential introduction of bias through a nesting effecting of previously discussed complex interpersonal relationships between evaluators and trainees though time. Written comments on EPA submissions were not assessed in the present study due to the inability to appropriately anonymize trainee and evaluator data. Similarly, it was not possible to extract whether an EPA assessment was directly or indirectly observed which may bias our data to a greater extent for cognitive EPA assessments which are more frequently observed indirectly. Lastly, a requirement of the local training program is ongoing EPA assessments for all clinical rotations, including those after the full demonstration of competence in all EPAs required by the RCPSC to graduate. This can be assumed to cause a slightly increased proportion of EPAs evaluated as “competence achieved” as trainees may have already demonstrated competence in the context assessed; thereby contributing to the aforementioned nesting observation induced bias.

Overall, we demonstrate ongoing barriers to high-quality trainee assessment and feedback for both procedural and cognitive assessments when implemented in a real-world context. We report a disproportionately large amount of EPA assessments denoting competence across all EPA types with low rates of complete O-SCORE utilization amongst evaluators. We also highlight the potential role that evaluator characteristics such as subspecialty training and type of practice play in CBME. This stresses the need for continued cycles of critical appraisal to improve the quality of CBD in Canada for the development of competent future physicians. We also emphasize the need for ongoing faculty and learner development to foster the culture change needed to shift the paradigm of viewing 1/5 or 2/5 as “bad” towards viewing these lower scores as teachable moments in progression of a trainee over time. Future directions include the formal validation of the O-SCORE or a modified scoring tool such as our proposed 3-point scale for the use in cognitive EPA evaluations. In this pursuit, structured prospective studies are required to minimize the influence of complex sources of bias utilizing standardized methodology such as mandated evaluator-initiated EPA assessments.

Abbreviations

CBME

Competence based medical education

CBD

Competence by design

RCPSC

Royal College of Physicians and Surgeons of Canada

EPA

Entrustable professional activity

O-SCORE

Ottawa Surgical Competency Operating Room Evaluation

SD

Standard Deviation

IQR

Interquartile Range

Authors' contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by MG, JC, and KIK. HL acts as the data custodian and is responsible for the integrity and access to the postgraduate medical education evaluations database from which data selected for inclusion and deidentified prior to distribution to the remainder of the study team. The first draft of the manuscript was written by JC and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Funding

No funds, grants, or other support was received.

Data availability

The datasets used and analyzed during the current study are available from the corresponding author on reasonable request.

Declarations

Ethics approval and consent to participate

This study was performed in line with the principles of the Declaration of Helsinki. The protocol for this study was approved by the University of Alberta Research Ethics Board (Ethics approval number: Pro00126104). A waiver of consent was granted by the University of Alberta Research Ethics Board given the administrative nature of the anonymized retrospective data and the inability to reasonably, feasibly, or practically obtain consent from all participants due to changes in contact information and the high volume of participants included.

Consent for publication

Not applicable. No identifying images, other personal details, or clinical details of participants that would compromise anonymity are presented in this manuscript.

Competing interests

Authors JC, HL, and JS have no relevant financial or non-financial interests to disclose. Author MG has received speaker fees from Pfizer and Janssen and participated in consultancy meetings with AbbVie, Takeda, Celltrion, Ferring and Pendopharm. Author KIK has received investigator-initiated funding from Pfizer; she has served on advisory boards or speaker for Abbvie, Janssen, Pfizer, Takeda, Celltrion.