An explorative study from the Norwegian Quality Register Gastronet comparing self-estimated versus registered quality in colonoscopy performance

Abstract

Background and study aims: The value of a colonoscopy quality assurance (QA) register may be questioned if it brings no new information on which to act for quality improvement, e. g. if self-assessed quality of colonoscopy performance correlates perfectly with registered performance.

Patients and methods: In this explorative study, 39 (33 Norwegian and 6 Swedish) out of 99 new endoscopists joining the Norwegian QA register Gastronet from 2008 to 2013 responded to an invitation to fill in a questionnaire for self-assessment of cecal intubation rate, polyp detection rate for polyps ≥ 5 mm (PDR-5 mm), withdrawal time, total examination time, and rates for severely painful and pain-free colonoscopies before receiving their first-time feedback of actually registered results from Gastronet. A linear regression analysis was applied to explore the correlation between experience level and quality of estimation.

Results: We included 2654 colonoscopies in our study. Endoscopists underestimated their cecal intubation rate (estimated 88.8 %, registered 93.1 %, P < 0.001), total procedure time (estimated 31.7 minutes, registered 37.2 minutes, P = 0.014), withdrawal time (estimated 9.8 minutes, registered 14.4 minutes, P = 0.006) and the rate of pain-free procedures (estimated 18.3 %, registered 24.5 %, P = 0.001). Pre-study colonoscopy experience was not correlated with estimated quality for any of the indicators.

Conclusions: Apart from overestimation of severely painful examinations, endoscopists most often underestimated their colonoscopy performance. Self-assessed quality of colonoscopy performance may not be a satisfactory substitute for systematic registration of quality and not sufficiently valid to be acted upon.

Introduction

Colonoscopy is currently regarded as the gold standard for diagnosis of colorectal conditions, including polyps and cancer. Demographic changes with an older population in many countries call for efforts to meet increasing demands for endoscopy services. Along with a steady increase in the number of colonoscopies performed, there has been a growing concern about the quality of colonoscopies. In recent years, European and American guidelines for quality of colonoscopy have been established giving both center leads and individual endoscopists a tool to assess quality of performance and service provided, ^{1 2} in addition to several quality assurance (QA) programs ^{3 4}.

To our knowledge, there have been no studies exploring the ability of individual colonoscopists to self-assess or guess the quality of their performance compared to actually measured performance results.

In Norway, the Gastronet QA program was launched in 2003 ⁴. Participation has been voluntary. Endoscopic retrograde cholangiopancreatographies (ERCP) (1,076 from 11 Norwegian center) and colonoscopies (15,423 from 25 Norwegian centers) were reported to Gastronet in 2014. In addition 3,123 colonoscopies from 3 Swedish clinical center and 5 Norwegian screening project sites were registered last year. Previously, gastroscopies were also reported but due to limited resources in the Gastronet secretariat and work load for the endoscopists, gastroscopy registration was ended. .

Both center leads and individual endoscopists receive feedback on different indicators capturing quality of colonoscopy (rate of completed colonoscopies, polyp detection and colonoscopies described as severely painful by the patient – in addition to the rate of procedures performed with sedation/analgesia and degree of patient satisfaction with the service). Registrations are paper-based using one form to be filled in by the endoscopist immediately after the colonoscopy and another to be filled in by the patient on the day after the procedure. Both forms are then returned to the Gastronet secretariat by mail.

The aim of this study was to explore to what degree endoscopists (both experienced and inexperienced) were capable of estimating their own colonoscopy performance. These self-estimates were then compared with the results from actually registered quality indicators in Gastronet.

Patients and methods

Endoscopists from Norway and Sweden who registered in Gastronet for the first time between 2008 and 2013 were eligible for this explorative study. Shortly after enrollment in Gastronet they were mailed a questionnaire and asked to estimate their colonoscopy performance on the basis of their colonoscopy experience so far. Those who responded and returned the questionnaire before they received individual feedback information from Gastronet for the first time were registered for this study.

We focused on the following four aspects of colonoscopy: rate of completed procedures, rate of polyp findings, duration of the procedure including insertion and withdrawal times, and subjective perception of pain reported by the patient by means of a four-point verbal rating scale (no, slight, moderate or severe pain). The patient filled in the questionnaire the day after the procedure and returned it to Gastronet in a prepaid envelope.

The endoscopists estimated the percentage of procedures during which colonoscopy was completed (cecal intubation rate [CIR]), the percentage of procedures during which polyps measuring at least 5 mm were found (polyp detection rate for polyps ≥ 5 mm [PDR-5 mm]), and how many minutes, on average, were needed to perform a full procedure and to withdraw the endoscope from the cecum to the anus while inspecting the colon mucosa for pathological findings. Total examination time and withdrawal time were restricted to diagnostic colonoscopies without therapeutic interventions. The insertion time from anus to cecum was calculated by subtracting withdrawal time (WT) from total procedure time. The endoscopists were also asked to estimate what percentage of their patients experienced severe pain and no pain, respectively, during the procedure. After returning the form with individual performance estimates, the endoscopists then received individual feedback on their performance results based on registrations in Gastronet. The results from self-assessment were then compared with registered performance data in Gastronet. We also explored whether differences were dependent on endoscopist experience or gender. Inexperienced colonoscopists were defined as having performed fewer than 300 colonoscopies.

The thresholds for good performance in our study followed international guidelines. American and European guidelines recommend a cecal intubation rate of ≥ 90 % ^{1 2}. During our study, American guidelines recommended an adenoma detection rate of 20 % (25 % for men and 15 % for women) ⁵. Because 80 % of colorectal polyps ≥ 5 mm have been shown to be adenomas ^{6 7}, the defined target in Gastronet was detection of polyps ≥ 5 mm of 25 % or more (20 %/0.8). In recently updated American guidelines, recommended adenoma detection rates have been increased to 30 % for male patients and 20 % for females ¹. The guidelines also recommend a withdrawal time in negative-result screening colonoscopy of ≥ 6 minutes. In our study, endoscopists were asked to estimate their individual mean withdrawal time for diagnostic colonoscopies without any therapeutic interventions. There are no international recommendations for the duration of total procedure time or insertion time. Likewise, international guidelines do not recommend standards for patient feedback on pain during colonoscopy. The endoscopists’ average rate of procedures with severe pain for the patient registered in Gastronet ranged from 13 % to 11.5 % in the last 5 years (13 % in 2010, 12 % in 2011, 11.8 % in 2012, 12.2 % in 2013 and 11.5 % in 2014). Therefore Gastronet recommends that endoscopists aim at a lowest possible rate with a maximum level of 12 %. To our knowledge, there are no recommendations for the rate of pain-free colonoscopies.

Statistical methods

Paired-samples t-test was applied to compare self-assessed with registered performance.

The width of the paired-sample t-test confidence intervals was used to assess the uncertainty in our estimates.

To explore the importance of colonoscopy experience for the ability to self-assess quality, we performed a linear regression for each indicator (cecal intubation rate [CIR], PDR-5 mm, total examination time, insertion time, WT, severe pain, no pain). The predictor variable was the estimated number of colonoscopies performed by the endoscopist during his/her career before entering Gastronet. The dependent or response variable was the difference between estimated and calculated value for each indicator. The presumption was that the differences between estimated and measured values might decrease with increasing endoscopist experience (number of performed procedures).

An independent two-sample t-test was applied to see if male and female endoscopists differ with regard to quality of self-assessment. All tests were two-sided, and P < 0.05 was considered statistically significant. All analyses were conducted with SPSS, version 21.

In order to evaluate the reliability of estimated compared with calculated (observed) values the intraclass correlation coefficient (ICC) was calculated for each quality indicator.

Ethics

The Regional Committee for Medical and Healthcare Research Ethics of the South Eastern Norwegian Health Board waived their need to evaluate the study protocol.

Results

In total, 99 endoscopists who registered for the first time in Gastronet between 2008 and 2013 were sent a questionnaire for estimation of their colonoscopy performance. Thirty-three (52 %) of the 63 Norwegian candidates responded. Twenty-five (40 %) did not respond and five (7.9 %) were erroneously invited because they already had been registered in Gastronet and previously obtained their Gastronet results (non-eligible for the study). Six (17 %) out of 36 Swedish candidate endoscopists responded (Table 1).

Table 1. Inclusion of endoscopists per May 2014.

	Norway	Sweden	Total
Invited, responded	33 (52 %)	6 (17 %)	39 (39 %)
Invited, no response	25 (40 %)	30 (83 %)	55 (56 %)
Not eligible (already having received Gastronet results)	5 (7.9 %)	0	5 (5.1 %)
Total	63	36	99

The participating endoscopists in this study had greatly varying pre-study experience defined by the number of colonoscopies performed. The endoscopist with the lowest level of experience had only performed 30 colonoscopies before entering Gastronet. The most experienced endoscopist estimated his pre-study experience at 5000 procedures. Twenty-two endoscopists had performed fewer than 300 colonoscopies before Gastronet registration. Sixteen endoscopists estimated their previous experience at 300 or more colonoscopies. One participant did not give information about previous experience.

The number of colonoscopies registered in Gastronet in this study varied between 30 and 170 procedures for a single endoscopist. The median number was 59 procedures per endoscopist. In total, 2654 procedures were included. The results from the paired-sample Student’s t-test are summarized in Table 2.

Table 2. Results of the paired-sample Student`s t-tests for the means of estimated and registered colonoscopy quality indicators for all included endoscopists.

	ESTIMATED					REGISTERED					t	P-value	mean, decrease/increase	lower 95 % CI	upper 95 %CI	ICC
	Mean	SD	Min.	Max.	Median	Mean	SD	Min.	Max.	Median
Cecal intubation (min)	88.8	5.5	70	95	90	93.1	4.4	83.6	100	93.8	–4.24	< 0.001	–4.26	–6.29	–2.22	0.036
PDR-5 mm (%)	16.3	12.5	5	70	15	20.8	9.3	3.2	54.8	19.6	–1.9	0.066	–4.46	–9.23	0.31	0.127
Total colon. time (min)	31.7	8,8	15	50	30	37.2	15.7	13.4	86.2	35.2	–2.58	0.014	–5.49	–9.79	–1.19	0.399
Insertion time (min)	21.7	7.9	10	35	20	23	8.7	8.8	37.3	23.2	–1.13	0.265	–1.35	–3.75	1.06	0.611
Withdrawal time (min)	9.8	2.9	3	15	10	14.4	9.8	3.8	49	12.6	–2.94	0.006	–4.62	–7.8	–1.44	0.013
Severe pain (%)	18.2	12.5	1	60	15	14.1	7.8	0	42.1	14.3	1.6	0.118	4.05	–1.07	9.17	0.163
no pain (%)	18.3	9.4	5	50	20	24.5	9.2	8.6	45	24.5	–3.5	0.001	–6.16	–9.73	–2.59	0.18

Abbreviations: Min., minimum; Max., maximum; SD, standard deviation; CI, confidence interval; ICC, intraclass correlation coefficient

Cecal intubation rate

Estimated CIR values ranged from 70 % to 95 % compared to 83.6 % to 100 % for registered CIR (Fig. 1 a) The mean estimated CIR was 88.8 %, compared to the registered CIR 93.1 %, P value < 0.001), Table 2 and Fig. 2. Only 26 participants estimated that they met the required 90 % level of cecal intubation rate, while 30 endoscopists fulfilled the requirements according to registered results.

Fig. 1.

 Scatterplots illustrating the relation between self-estimated quality and registered results for the various quality indicators for each individual endoscopist. Each circle represents one endoscopist. The x-axis displays that endoscopist’s guessed value, the y-axis shows the registered value. Circles on or very close to the 45°-line represent endoscopists who correctly guessed their quality. Blue circles represent endoscopists with less than 300 colonoscopies before registration in Gastronet and the red circles display more experienced endoscopists with 300 or more colonoscopies. a Cecum intubation rate. b Polyp detection rate for polyp ≥ 5 mm. c Rate of severely painful colonoscopies. d Withdrawal time

Fig. 2.

 Bar chart showing the endoscopists’ mean values for self-assessed and registered performance in Gastronet for each quality indicator (P values from paired-samples t-test) (39 endoscopists).

Polyp detection

The estimated PDR-5 mm detection rate ranged from 5.0 % to 70 %, and 3.2 % to 54.8 % for registered PDR-5 mm (Fig. 1 b). The participants estimated their polyp detection rate (PDR-5 mm) to be slightly worse than registered (mean estimated 16.3 %, mean registered 20.8 %) but the difference did not reach statistical significance, P = 0.07. The target of PDR-5 mm of 25 % or higher was met by 11 endoscopists (29 %).

Pain during colonoscopy

The estimated proportion of severely painful colonoscopies ranged from 1 % to 60 % and 5 % to 50 % for pain-free procedures. The corresponding registered results were 0 % to 42 % and 8.6 % to 45.0 %, respectively (Fig. 1 c).The rate of severely painful colonoscopies was estimated slightly higher than actually registered, but not statistically significant (mean estimated 18.2 %, mean registered 14.1 %, P = 0.118). Conversely, the endoscopists estimated their rate of pain-free procedures to be rather low (mean estimated 18.3 %, mean registered 24.5 %, P = 0.001).

Duration of procedure

The range for estimated total procedure time was from 15 to 50 minutes and 3 to 15 minutes for withdrawal time. The range for registered results in Gastronet was from 13.4 to 86.2 minutes for total examination time and 3.8 to 49 minutes for withdrawal time. (Fig. 1 d). The mean estimated insertion time (21.7 min) was very close to the registered value (23.0 min), P = 0.27. In contrast, the endoscopists underestimated the time they used for withdrawal and inspection in diagnostic procedures. The mean estimation was 9.8 minutes and the registered result 14.4 minutes, P = 0.006. Accordingly, total examination time was also underestimated (estimated 31.7 minutes and registered 37.2 minutes, P = 0.01). Fig. 2 displays a bar chart summarizing results for each quality indicator.

Colonoscopy experience

The precision of self-estimates did not improve with increasing pre-study colonoscopy experience. In the linear regression model, none of the quality indicators reached significance level (lowest p-value 0.17 and highest 0.74, regression line and table not shown). In Fig. 1 a, 1 b, 1 c, and 1 d, four scatterplots for different quality indicators depict the range of self-assessment quality for inexperienced (blue circles) and experienced endoscopists (red circles).

Endoscopist gender

Female endoscopists estimated their insertion time to be 5.5 minutes shorter than registered values whereas their male colleagues estimated their insertion time to be 0.6 minutes longer than the registered values (95 %CI 1.2 minutes – 10.9 minutes, P = 0.02). For all other quality indicators, there were no differences in self-assessment between male and female endoscopists.

Intraclass correlation coefficient

The ICC was low for all quality indicators. The hightest ICC was for insertion time (0.611) and the lowest ICC was 0.013 for withdrawal time. The results are shown in Table 2. This indicates that the reliability of estimated compared to observed values was moderate for insertion time and low for all other quality indicators.

Discussion

Quality registers aim to provide information that is 1) not available otherwise and 2) valid for responsible persons to act on to improve or maintain quality. This explorative study suggests that self-assessed quality of colonoscopy performance is not a satisfactory substitute for systematic registration of quality and not sufficiently valid to be acted upon.

There was great variation in the ability to self-assess colonoscopy performance with a tendency for endoscopists to underestimate the quality of their performance. One exception was self-assessment of severely painful colonoscopies which tended to be higher than registered.

In a review by Gordon ⁸ the author concluded that the validity of self-assessed performance was found to be low to moderate. It did not improve with time in training programs and it was minimally influenced by external feedback. Another review article by Davis et al. ⁹ also concluded that physicians had a limited ability to accurately self-assess. In this review, 13 out of 20 comparisons between self- and external assessment demonstrated little, no or an inverse relationship and only seven demonstrated positive associations. The worst accuracy in self-assessment was found among physicians who were the least skilled and those who were the most confident.

Cecum intubation

The mean self-assessed cecal intubation rate in our study was 4.3 % less than registered CIR (self-assessed 88.8 %, registered 93.1 %, P < 0.001). More endoscopists accomplished the desired level of CIR of ≥ 90 % (30 endoscopists) than shown by self-estimated CIR (26 endoscopists). Incorrect self-estimation ranged from mild overestimation (one endoscopist estimated CIR to 90 % while it was registered as 85 %) to gross underestimation (one endoscopist with self-estimated CIR of 70 % which was registered as 93 %). Several studies have focused on learning curves for CIR among endoscopist trainees ^{10 11 12 13}. The number of colonoscopies needed to accomplish a CIR of 85 to 90 % ranged from 150 to 280 procedures. None of the studies implemented a self-estimation by the endoscopist. From our data we can conclude that the participating endoscopist underestimated their completion rate and the capability to estimate the individual CIR varies greatly among endoscopists as depicted in Fig. 1 a. In Gastronet, cecal intubation is registered as successful independent of whether the endoscopist needed help from a more experienced colleague or not. Therefore the CIR for endoscopists with very little pre-study experience might reflect a well-functioning master/apprentice cooperation rather than genuine unaided caecum intubation of the trainee.

Polyp detection

In our study, endoscopists showed a tendency to underestimate polyp detection. The mean self-estimation was 16.3 % PDR-5 mm compared to 20.8 % registered (P = 0.066). Similar to CIR, the correlations between estimated and registered PDR-5 mm values for different endoscopists vary greatly as demonstrated in Fig. 1 b. In a study by Ansell et al., endoscopists of all experience levels from novice to expert performed polypectomy in simulated colonoscopy ¹⁴. Each procedure was assessed by a structured assessment form both by the endoscopist himself or herself and two expert assessors. They concluded that the correlation between assessors’ scores and self-assessment scores was weak.

Different approaches have been explored to improve diagnostic methods and treatment of colorectal polyps. Gupta et al. tested a polypectomy competence assessment tool (Direct observation assessment tool) ¹⁵. Dawn et al. established a conversion factor to estimate the adenoma detection rate from the polyp detection rate ¹⁶. Both authors concluded that their tools can contribute to improve the quality of colonoscopy services for treatment of polyps. Self-estimation of quality by endoscopists was not implemented in these two studies.

Pain during colonoscopy

By self-assessment, the endoscopists tended to overestimate the proportion of severely painful and underestimate painless colonoscopies reported by their patients. The mean estimate for severe pain was 18.2 % compared to 14.1 % reported (P = 0.12). The self-estimated proportion of pain-free procedures was 18.3 % compared to 24.5 % reported by the patients as being pain-free (P = 0.001). In line with the other indicators, there was a wide spectrum of estimations ranging from overestimation, good estimation approximating registered results to underestimation as illustrated in the Fig. 1 c. In a study by Heuss et al., 222 patients scheduled for outpatient colonoscopy rated their discomfort related to the procedure ¹⁷. Similar to our study, they concluded that estimation of discomfort for the patient by health personnel (both endoscopists and nurses) is difficult and poorly correlated to that reported by the patients themselves. Both doctors and nurses tended to overestimate the patients’ discomfort. However, in 9 % of cases the level of patient discomfort was considerably underestimated by at least one member of the team. In our study, 16 endoscopists (41 % of all) estimated their rate for severely painful colonoscopies to be less than the registered rate in Gastronet. For twelve of them (31 % of all endoscopists) the registered rate for severely painful procedures was 5 % or more above the self-estimated rate. This means that almost one third of the endoscopists reckon their performance with regard to pain is better than reported by the patients.

The wide range of estimation of quality in both studies reflects the high grade of subjectivity related to assessment of pain. In the context of clinical routine endoscopy, there is no direct objective measurement of pain. Consequently, from our point of view, subjective feedback from the patient himself/herself should define the amount of discomfort or pain perceived.

Procedure duration

The endoscopists in our study used more time both for the entire procedure and for withdrawal than reflected in their self-estimations of average time spent. The registered mean for total examination time and withdrawal time was significantly longer than estimated (total time 31.7 min estimated and 37.2 min registered, P = 0.01. WT 9.8 min estimated and 14.4 min registered, P = 0.006). The estimated insertion time (mean 21.7 min) met quite well the registered value (mean 23.0 min), P = 0.27. We can conclude that the total procedure time is longer than estimated because the endoscopists use more time on withdrawal than they estimate themselves. American guidelines recommend a WT of at least 6 minutes in purely diagnostic procedures ¹. According to those guidelines, the endoscopists in our study were well within standards. Spending twice as much time on withdrawal and inspection than estimated by the endoscopists themselves may suggest an overzealous attitude by a highly motivated endoscopist embarking on a new QA program. This, however, goes beyond the scope of the present study.

Colonoscopy experience

One might expect that increasing colonoscopy experience would make it easier for an individual endoscopist to make a good guess about his/her own colonoscopy quality. But we did not see evidence of that in our linear regression model with a limited number of endoscopists. Thus, we cannot conclude that a higher number of pre-study colonoscopies (i. e. endocopist experience) may reduce the difference between self-assessed and registered result. Studies with larger numbers of endoscopists are needed to analyze the correlation between experience level and self-assessment quality. Self-assessment of personal quality of performance, however, appears to have very limited value for and impact on QA work regardless of level of experience.

Endoscopist gender

Apart from a significant underestimation of insertion time by female endoscopists compared to male endoscopists, there was no significant difference between male and female endoscopists with regard to CIR, PDR-5 mm, total examination time, WT, or rates for severely painful and pain-free colonoscopies. Endoscopist gender, therefore, does not appear to be a major issue in self-assessment of colonoscopy performance.

Limitations

The response rate from invited endoscopists was low at only 39 %. Accordingly the number of participating endoscopists was low with only 39 participants. Gastronet did not investigate why the response rate was low. In Norway 25 out of 58 eligible endoscopists (43 %) and in Sweden 30 out of 36 eligible endoscopists (83 %) did not respond to the invitation. Participation rates in the different centers ranged from only three out of 14 invited endoscopists (21 %) in a tertiary center in Sweden to all out of six invited (100 %) in a center in South Norway. We do not know why participation was low, particularly in Sweden.

We can only speculate whether the reasons for this might be high workload, embarrassment about giving a wrong estimation, a negative attitude toward quality improvement work, worry about being confronted with personal suboptimal performance or other unknown reasons. Analysis of non-participation goes beyond the scope of a quality assurance initiative. Given the high non-participation rate, we cannot exclude the possibility of selection bias affecting our results. This clearly represents a weakness in our study.

Apart from CIR, all quality indicators show rather wide 95 % confidence intervals in the paired-sample t-test (Table 2). Indicators with a P value approaching the 0.05 % significance level may reach significance in a larger endoscopist cohort, as for example PDR-5 mm with a P value of 0.066.

Conclusions

Endoscopists do not accurately estimate their own performance with regard to several colonoscopy quality indicators. There is wide variation ranging from overestimation, to good estimation, to underestimation. Experience level was not correlated with the quality of self-estimation in our data but that might be due to the low number of participants. Quality of self-estimation does not differ with gender of the endoscopist.