Self-assessment of Competence in Endoscopy: Challenges and Insights

Abstract

Background

Endoscopists use self-assessment to monitor the development and maintenance of their skills. The accuracy of these self-assessments, which reflects how closely one’s own rating corresponds to an external rating, is unclear.

Methods

In this narrative review, we critically examine the current literature on self-assessment in gastrointestinal endoscopy with the aim of informing training and practice and identifying opportunities to improve the methodological rigor of future studies.

Results

In the seven included studies, the evidence regarding self-assessment accuracy was mixed. When stratified by experience level, however, novice endoscopists were least accurate in their self-assessments and tended to overestimate their performance. Studies examining the utility of video-based interventions using observation of expert benchmark performances show promise as a mechanism to improve self-assessment accuracy among novices.

Conclusions

Based on the results of this review, we highlight problematic areas, identify opportunities to improve the methodological rigor of future studies on endoscopic self-assessment and outline potential avenues for further exploration.

Introduction

Assessment guides the development and maintenance of competence in medicine (1). Although gastrointestinal endoscopy trainees are commonly assessed by an external rater (2,3), self-assessment has been proposed as a means by which complementary data can be collected (4). Self-assessment, defined as “a personal evaluation of one’s professional attributes and abilities against perceived norms” (5), has implications as a meta-cognitive skill that underpins continual skill development (6–8). In particular, self-assessment has a role in the management of one’s own learning (9), as one study demonstrated that improved self-assessment skills led to more fruitful self-regulated learning (10). Given the impact of self-regulated learning in clinical contexts (11), the application of self-assessment requires careful consideration.

Specifically, self-assessment must be accurate—that is, one’s self-assigned rating should correspond to one’s skill level and align with externally assigned ratings (12). Current evidence, however, strongly suggests that health professionals cannot accurately assess their own performances with respect to clinical skills (e.g., diagnostic ability) and cognitive skills (e.g., general medical knowledge) (13–15). Conversely, there is uncertainty about whether self-assessment is more accurate for procedural skills. In particular, a narrative review of surgeons found mixed results regarding procedural self-assessment accuracy (16).

In gastrointestinal endoscopy, the importance of self-assessment has been acknowledged by credentialing bodies, such as the Joint Advisory Group on Gastrointestinal Endoscopy (JAG), which recommends that trainees incorporate self-assessment practices into their training (17). Additionally, it has been emphasized as an important component of maintenance of certification by professional organizations, such as the Royal College of Physicians and Surgeons of Canada, to enable identification of opportunities to enhance competence through future learning activities (18). To date, however, the evidence surrounding endoscopic self-assessment has yet to be thoroughly examined. Although there are systematic reviews of assessment in endoscopy (3,19), self-assessment was addressed only briefly. In this narrative review, we provide a summative, critical examination of the endoscopic self-assessment literature.

Current State of Evidence of Endoscopic Self-Assessment

There is a growing evidence base on endoscopic self-assessment as we identified seven studies on the topic (see Table 1) (20–26). One of the first studies that presented data on endoscopic self-assessment accuracy was published by Vassiliou et al. in 2010 (20). Although the primary focus of the paper was the validation of two technical skills assessment tools, the Global Assessment of Gastrointestinal Endoscopic Skills for upper endoscopy (GAGES-UE) and for colonoscopy (GAGES-C), self-assessment accuracy data was presented as a secondary outcome (20). The authors reported good agreement between external and self-assessed scores using both tools. Another study examined self-assessment in the context of simulated colonoscopic polypectomy (21). The authors reported an overall weak correlation between externally and self-assessed scores using the Direct Observation of Polypectomy Skills (DOPyS) tool, among a group that included endoscopists of novice, intermediate, advanced and expert experience levels (27). When the analysis was stratified by experience level, advanced and expert endoscopists were shown to have relatively more accurate self-assessment, albeit they were still considered “inaccurate.” Of note, the authors found that novices and intermediates tended to underestimate their performances, while advanced and expert endoscopists tended toward overestimation. The two key findings from this study, inaccurate overall self-assessment and underestimation of performance, were also found in the study by Moritz et al. on self-assessed colonoscopy quality metrics (22). In particular, the primary analysis revealed that endoscopists were inaccurate as they underestimated their performance on several parameters of colonoscopy quality that were recorded in a quality assurance registry, including cecal intubation rate and withdrawal time. Furthermore, there were no significant differences when accounting for endoscopic experience or for gender, although the authors remark that their study may have been underpowered to detect these differences.

Table 1.

Summary of the self-assessment literature in gastrointestinal endoscopy

First author and year	Methodology			Findings
	Endoscopic setting	Endoscopist sample	Endoscopic procedure(s)	Overall accuracy of self-assessment	Direction of self-assessment bias (if present)	Effect of endoscopic experience on self-assessment accuracy
Vassilou (2010)	Clinical	Adult endoscopists	Colonoscopy, gastroscopy	Accurate	N/A	N/A
Ansell (2014)	Simulated	Adult	Colonoscopy	Inaccurate	Overestimation	Improved
Moritz (2016)	Simulated	Adult	Colonoscopy	Inaccurate	Underestimation	No change
Vyasa (2016)	Simulated	Adult	Sigmoidoscopy	Accurate	Overestimation	N/A
Scaffidi (2018a)	Clinical	Adult	Colonoscopy	Accurate	Overestimation	Improved
Scaffidi (2018b)	Clinical	Pediatric	Colonoscopy	Accurate	Overestimation	Improved
Scaffidi (2019)	Simulated	Adult	Gastroscopy	Accurate	Overestimation	N/A

N/A, not applicable

While the previous studies suggest a minimal to negligible effect of endoscopic experience on self-assessment accuracy, two recent studies indicate otherwise. A 2018 study by Scaffidi et al. that aimed to evaluate self-assessment accuracy of novice, intermediate and experienced endoscopists found that there was a positive relationship between experience level and self-assessment accuracy of clinical competence (24). Although there was an overall moderate agreement between externally and self-assessed scores on the Gastrointestinal Endoscopic Colonoscopy Assessment Tool (GiECAT), a procedure-specific assessment tool, experienced endoscopists were significantly more accurate in their self-assessments compared with novices (28). Furthermore, experienced endoscopists tended to underestimate their performances, while novices tended to overestimate themselves. A subsequent study by the same group in the setting of pediatric clinical colonoscopies yielded similar results (25). Specifically, compared with novices, experienced endoscopists had more accurate self-assessments using the Gastrointestinal Endoscopy Competency Assessment Tool for Pediatric Colonoscopy (GiECAT_KIDS), wherein novices overestimated their performances and experienced endoscopists underestimated (29). Taken together, these studies provide compelling evidence that experience can positively influence self-assessment accuracy. This finding has a strong theoretical basis, which will be discussed in a later section.

A consistent trend in the extant literature is that novice endoscopists demonstrate inaccurate self-assessment, a deficiency that two studies sought to remedy using video-based feedback (23,26). The first, a trial published by Parth et al. in 2016, involved general surgery residents self-assessing their ability to perform a simulated screening colonoscopy using procedural metrics (e.g., time to cecum) generated by a virtual reality simulator (23). After completing a pretest, participants were randomized into one of three arms: video review of their own simulated procedures; video review of an expert completing the same procedure; or additional practice on the simulator (instead of video review) as a control. Participants then completed 10 additional simulated practice cases after the intervention, followed by a posttest on the same initial simulated case. The authors reported a small improvement in self-assessment accuracy among participants who watched the expert benchmark video. The second, a study by Scaffidi et al., used a similar methodology, although the simulated procedure was esophagoduodenoscopy (EGD) and participants, while all novice endoscopists, were recruited across different specialties and varying levels of training, which included medical students, residents and staff physicians (26). After completing a simulated EGD procedure as a pretest, participants were randomized to one of three interventions: video review of their own simulated EGD case; video review of a benchmark (i.e., expert) performance of the simulated EGD case; or access to both videos for review. Participants then completed two posttests: the same simulated EGD case as the pretest, which determined skill retention, and a novel EGD case, which determined transfer. Self-assessments were completed during all testing using the global rating scale component of the GiECAT (28). The authors found that both the benchmark and combination interventions led to improved self-assessment accuracy, although the former group only demonstrated improvement on the retention test, while the latter group only on the transfer test, suggesting that combination video review leads to more sustained improvements in self-assessment accuracy.

Challenges with Endoscopic Self-Assessment

Despite the growing evidence base regarding endoscopic self-assessment, a summative interpretation of existing studies is limited by several key methodological deficiencies. The first pertains to the assessment metrics utilized. Studies aiming to investigate self-assessment should ensure that the assessment metrics employed have adequate validity evidence. Much of the current literature utilized simulator metrics (23), quality metrics and/or procedural endpoints (e.g., procedure duration) (22) as surrogate measures of endoscopic competence. Although these are easy to calculate and interpret, there is mixed evidence regarding their validity as markers of endoscopic competence (3,19). Moreover, these often singular-measure approaches fail to holistically capture the multifaceted dimensions of endoscopic competence, including those that relate to cognitive (e.g., pathology recognition), technical (e.g., scope navigation) and integrative (e.g., communication) skills (2). There are a number of direct observational procedural assessment tools with strong validity evidence that have been recommended for assessment of endoscopic competence (2,19), such as the DOPyS (21), the GiECAT (24,26), or the GiECAT_KIDS (25), which have been used in the more recent studies of higher methodological quality.

Second, there are several statistical techniques that have been used inappropriately in previous studies of endoscopic self-assessment. In particular, linear correlation measures, such as Spearman’s rho (21), cannot be used to capture agreement between external- and self-assessors because they are prone to systematic variations in the data, such as chance agreement between assessors (16,30). Another, suboptimal method used is the scatterplot of external and self-assessed scores (22), which, while not necessarily incorrect, fails to adequately detect systematic subtleties (i.e., overestimation and underestimation) between the two sets of scores. Based on the method comparison literature (31), we recommend the following three techniques: Bland–Altman plots, which can visualize systematic differences, such as overestimation or underestimation, by plotting the mean differences between the two sets of raters against the mean values of the scores: the intraclass correlation coefficient (ICC), which quantifies agreement between two or more raters while correcting for chance agreement, and absolute difference scores, which involve the calculation of the absolute differences between external- and self-assessors that can be used to determine if there are differences between groups of endoscopists (e.g., varying experience levels). These techniques were utilized in three of the aforementioned studies (24–26).

Insights Into Endoscopic Self-Assessment

To better understand the findings from the endoscopic self-assessment literature, we consider two relevant concepts from the psychological literature. The first is the Dunning–Kruger effect, which is a well-described psychological phenomenon, wherein those who are least competent overestimate their skill level due to their lack of awareness (32). In terms of endoscopic self-assessment, this phenomenon explains why most of the studies we identified demonstrate that novice endoscopists are the least accurate self-assessors (21,23–26) and why several studies found that novices tend to overestimate their competence (23–26). Furthermore, the Dunning–Kruger effect may provide a mechanistic explanation for the effectiveness of benchmark video-based interventions in improving self-assessment accuracy among novices. Accurate self-assessment requires insight into one’s own performance, an awareness of the performance of others and a capacity to reflect on both measures and make a judgment (33). Observation of experts may provide novices with realistic standards for assessing their own performance despite their inexperience (26).

Another possible explanation for inaccurate self-assessment is impression management, whereby individuals intentionally present a more favorable view of themselves (34). A study of self-assessment in oral surgery found that surgeons who inaccurately overestimated their performance also had higher impression management scores, indicating that they had a greater tendency to deliberately distort responses to present a more positive image of themselves (35). This framework has been used to explain why endoscopists, especially novices, may have inaccurate self-assessments that typically involve overestimation as they intend to create a favorable perception of their performances (21).

Implications for Endoscopic Training, Practice and Future Studies

Self-assessment is important for several reasons. It enables the identification of one’s weakness to help set appropriate learning goals, enable one to self-limit in areas outside of one’s scope of competence practice and to set realistic expectations for oneself. Finally, self-assessment is an integral component in the process of self-regulated learning, wherein one’s own learning is guided by meta-cognition (i.e., awareness and understanding of one’s own thought processes). Self-regulated learning has important implications for physicians both in training and in practice as skill development continues throughout one’s entire career. Research has highlighted the importance of self-assessment in relation to self-regulated learning. A recent paper summarizing four meta-analyses (19 studies) found that self-assessment practices had a considerable impact on individuals’ self-regulated learning, with effect sizes ranging from 0.23 to 0.65 (36). Furthermore, the model of self-regulated learning proposed by Winne and Hadwin asserts that self-assessment is essential in that it enables learners to ensure that they are on target with their learning objectives (37).

In general, we found mixed evidence regarding the accuracy of endoscopic self-assessment, with three studies reporting an overall moderate accuracy for endoscopists (20,24,25) and two studies reporting inaccurate self-assessment (21,22). When stratified by endoscopic experience, however, the primary at-risk group for inaccurate self-assessment are novices as they not only lack experience but are unaware of the extent to which they are unskilled (32). Although novice endoscopists demonstrate a clear inability to accurately self-assess, existing evidence is less clear regarding experienced endoscopists. Specifically, three studies demonstrated that more experienced endoscopists had better self-assessment accuracy as compared with inexperienced individuals (21,24,25). It is unclear, however, what threshold of endoscopic experience confers this capacity for improved self-assessment. While two of the aforementioned studies found that experienced endoscopists, who completed more than 1000 colonoscopies, were most accurate in their self-assessments, there was no clear trend in self-assessment accuracy for intermediate endoscopists, who completed between 50 and 500 colonoscopies (24,25).

In terms of endoscopic practice and training, we note several implications. First, endoscopists of any experience level should avoid the use of quality indicators and/or simulator-based metrics as the basis for self-assessment as these lack strong evidence of validity for endoscopic competence (3,19). Additionally, such singular measures provide little informative feedback to help pinpoint deficiencies (2). Instead, tools that focus on direct observation of skills (e.g., DOPyS and GiECAT) should be implemented for self-assessment. One way to implement self-assessment of endoscopic competence is through the use of video recordings, which have been shown to have good validity evidence (38). Videos can be recorded during the procedure and assessed at a later time by an external- and/or self-assessor, thus allowing more time for reflection on performance. More experienced endoscopists, who generally demonstrate better self-assessment accuracy, can incorporate self-assessment into their practice to help self-identify their learning needs to create individualized plans for their continuing professional development. They may wish, however, to monitor their initial assessments to ensure that they are accurate and can potentially engage in courses that provide structured assessment and feedback (39). Educators leading endoscopy-related professional development activities can also potentially use self-assessment data to tailor educational content to more closely match participants’ learning needs.

Conversely, novice endoscopists may require targeted attention with respect to self-assessment. In particular, video-based interventions can positively impact self-assessment accuracy. Two included studies found that novices who watched a benchmark video of an expert perform the same endoscopic procedure, before their own performance, had improved self-assessment accuracy (23,26). Video interventions targeting self-assessment accuracy have also demonstrated effectiveness in other skill domains, such as surgery (40).

External feedback from an expert can also play a key role in regard to self-assessment accuracy as the views of others can be incorporated to create the self-awareness necessary for learning. Eva and Regehr suggest that self-improvement requires not only reliable external feedback but also deliberate efforts to reflect on that feedback so that it becomes meaningful (41). Additionally, the combination of external feedback and self-assessment appears to be synergistic; a meta-analysis found improved learning when self-assessment was combined with feedback as opposed to self-assessment alone (42). However, literature has shown that the perceived quality of feedback has an impact on the likelihood that individuals will use the feedback to inform their self-assessment and improve their practices (43). It is important for endoscopic trainers to consider the manner in which novices engage in self-assessment and receive feedback. In line with the Dunning–Kruger phenomenon discussed earlier, less experienced individuals lack accurate insight into their performances and, therefore, tend to overestimate their awareness (32). This may also affect novices’ perception of feedback. An overconfident novice, for example, may dismiss negative feedback from an external assessor as unreliable. On the other hand, feedback delivered from a clear position of beneficence has been shown to be more likely perceived as worthy of attention (7). To ensure that feedback is understood and acted upon, endoscopic trainers should consider using informed self-assessment, wherein they review self-assessment data with trainees to foster self-reflection and help them to develop a more accurate perception of their abilities and learning deficits (7,44). This will, ultimately, enable them to become more self-aware and nurture their ability to self-direct their own learning.

There is a growing interest in the use of “objective” measures of performance, such as cecal intubation rate. While such quality metrics are attractive, there is evidence indicating that reliance on singular end-point measure may have unintended consequences as endoscopists may “game” certain metrics while disregarding others, which can hinder comprehensive feedback (45). A study by Razzak et al., which examined the impact of monitoring both adenoma detection rate and cecal withdrawal time, found that monitoring only withdrawal time led to fewer adenomas removed per examination (46). It is, therefore, unclear whether this strategy should be used among trainees, as their inexperience may potentially even exacerbate the unintended effect of “gaming” metrics. In practice, it has been suggested that a combination of metrics is preferable to provide a more comprehensive picture of performance (47).

There are opportunities to improve future research in the area of endoscopic self-assessment. First, the assessment measures used should have strong evidence of validity as evaluated through the lens of a validity framework, such as the one described by Messick (48,49). A recent systematic review used Messick’s framework to evaluate the validity evidence of methods for certification in colonoscopy (3). Several assessment tools, such as the DOPyS and the GiECAT, used in the more recent, higher-quality studies of endoscopic self-assessment (24–26), have strong evidence of validity. Second, appropriate measurement of self-assessment accuracy should be used. Specifically, we suggest three statistical techniques from the method comparison literature—the ICC; Bland–Atlman plots and absolute difference scores—as these are better suited to detect systematic differences in agreement among individuals (31). Finally, moderators of self-assessment, such as experience level (24,25), should be carefully considered in any investigation as the current findings demonstrate that they have an impact on self-assessment accuracy.

In terms of future studies on endoscopic self-assessment, we suggest three areas for exploration. First, a systematic review that quantitatively summarizes endoscopic self-assessment accuracy would provide an objective means by which to determine if endoscopists are accurate. Second, studies could further explore the impact of video interventions on self-assessment accuracy, for example, through the integration of benchmark videos into existing simulation-based training curricula aimed at novice endoscopists (50–55). Finally, future studies should evaluate the theoretical underpinnings of self-assessment accuracy, such as the measurement of impression management, which has been conducted for other procedures (35).

Conclusions

Self-assessment is an important meta-cognitive skill in the development and maintenance of procedural competence. In this narrative review, we examined the extant literature on endoscopic self-assessment through a summative, critical lens to highlight implications for training, practice and research. In general, we found that novice endoscopists are most inaccurate in their self-assessment, while experienced endoscopists are more accurate. Based on the current literature, we make several recommendations for training and practice, which includes use of direct observation assessment tools with strong validity evidence, use of video-based interventions to improve self-assessment accuracy (especially among novices) and incorporation of self-assessment into practice and training.

Funding

C.M.W. holds a Career Development Award from the Canadian Child Health Clinician Scientist Program and an Early Researcher Award from the Ontario Ministry of Research, Innovation and Science. The funders had no role in the design and conduct of the review, decision to publish and preparation, review, or approval of the manuscript.

Conflicts of Interest

R.K. has received research funding from AbbVie, Ferring Pharmaceuticals, and Pendopharm. S.C.G. has received research funding from AbbVie and Janssen and personal fees from AbbVie, Takeda and Ferring, and is owner of, and holds shares, Volō Healthcare. All other authors have no conflicts of interest to disclose.