Quality evaluation through self-assessment: a novel method to gain insight into ERCP performance

Abstract

Background

The American Society for Gastrointestinal Endoscopy Committee on Outcomes Research has recommended monitoring nine endoscopic retrograde cholangiopancreatography (ERCP)-specific quality indicators for quality assurance in ERCP. With the development of a self-assessment tool for ERCP (Rotterdam Assessment Form for ERCP—RAF-E), key indicators can easily be assessed.

Objective

The aim of this study was to test in daily practice an easy-to-use form for assessment of procedural quality in ERCP and to determine ERCP quality outcomes in a tertiary referral hospital.

Design

This was a prospective study carried out in a tertiary referral hospital. In January 2008, a quality self-assessment programme was started. Five qualified endoscopists participated in this study. All ERCPs were appraised using RAF-E. Primary parameters were common bile duct (CBD) cannulation rate and procedural success. The indication was classified and procedural difficulty was graded; success rates of therapeutic interventions were measured for all different difficulty degrees.

Results

A total number of 1691 ERCPs were performed. 1515 (89.6%) of these were appraised using RAF-E. Median CBD cannulation success rate was 94.1%. Successful sphincterotomy was accomplished in almost all patients (median 100%; range 98.2–100%). Stent placement was successful in 97.8% and complete stone extraction, if indicated, was achieved in 86.8%.

Conclusions

Quality indicators for ERCP can be measured using the Rotterdam self-assessment programme for ERCP. Outcome data in ERCPs obtained with this RAF-E provide insight into the quality of individual as well as group performance and can be used to assess and set standards for quality control in ERCP.

Background

In endoscopic procedures, the awareness on quality has increased over the last decade. Important landmarks are documenting whether the intended outcome of the procedure was achieved under the restriction of acceptable side effects, patient comfort and costs. Endoscopic retrograde cholangiopancreatography (ERCP) is a complex and challenging procedure for diagnosis and treatment of biliary and pancreatic disorders. It carries a relatively high risk of complications, and thus, assurance of quality is pivotal.^1–3

However, monitoring and enhancing the quality of endoscopic procedures is impossible without methods to register specific parameters. Therefore, the American Society for Gastrointestinal Endoscopy (ASGE) Committee on Outcomes Research has defined a set of quality indicators for ERCP.⁴ These quality indicators roughly fall in three separate domains: pre-procedural quality indicators (appropriateness of indication), procedural quality indicators (ie, common bile duct (CBD) cannulation rates and therapeutic success) and post-procedural outcomes (ie, documentation and complications). There have been several publications on complications as a marker of quality,^5–9 but procedural success or patient-related outcomes are less often described.¹⁰

In addition, self-assessment of procedural performance in ERCP might enhance quality by stimulating active reflection on one's actions. Questions on topics such as individual versus group performance, personal performance level compared with that of colleagues and how to learn from mistakes made can be answered through proper self-assessment. Furthermore, specific objective outcomes such as CBD cannulation rate can easily be calculated. In order to gain insight into quality of ERCPs, especially focusing on endoscopists’ performance, we developed a self-assessment programme for ERCP. A similar method has proven its value for colonoscopy.¹¹ Based on this study, as well as on previously validated assessment tools, the Rotterdam Assessment Form for ERCP (RAF-E) was developed (figure 1).^12–14

Figure 1.

The Rotterdam Assessment Form for endoscopic retrograde cholangiopancreatography (ERCP). PSC, primary sclerosing cholangitis.

The aim of this study was to test in daily practice an easy-to-use form for assessment of procedural quality in ERCP. A major drawback of an easy-to-use form to be completed after every procedure without taking up too much time is the number of parameters that can be measured. Second, we aimed to determine ERCP quality outcomes in a tertiary referral centre.

Methods

From January 2008 to December 2011, a prospective, single centre evaluation of ERCP performance was carried out in the Erasmus MC, University Medical Center in Rotterdam, the Netherlands. A quality assessment programme was carried out through the use of self-assessment. All staff members of the department performing ERCP participated. All ERCP procedures performed by these five qualified endoscopists were included in the analysis. Both scheduled and emergency procedures were included.

Participants completed RAF-E after each ERCP. The form, as shown in figure 1, contains an objective part and a subjective part. Objective parameters include indication, difficulty degree adapted from Schutz's classification (table 1),¹⁵ previous failed attempts at cannulation in a different centre and the presence of a ‘virgin’ papilla or previous sphincterotomy, as well as CBD cannulation success and success of various therapeutic procedures. In the subjective section, endoscopists are asked to grade their performance on a Visual Analogue Scale. After each completion of a set of 10 assessment forms, an improvement plan can be formulated based on the Osborn–Parnes Creative Problem Solving Process as used in other fields.¹⁶ The subjective scores are not taken into account in the statistical analysis, nor the outcome of the improvement plan. The value of the subjective assessment lies in creating self-awareness to enable self-reflection on performance rather than in providing evidence for quality measurements. To exclude registration bias, all non-self-assessed cases were evaluated as well. The number and outcome of these were studied by checking reports from the endoscopy database and assessed according to the same RAF-E criteria.

Table 1.

Degrees of difficulty based on Schutz's classification¹⁵

Difficulty degree	Biliary procedures	Pancreatic procedures
Grade 1	▸ Diagnostic cholangiography ▸ Biliary cytology ▸ Stone extraction ≤10 mm ▸ Dilatation of stenosis/stent placement/nasobiliary drain in extrahepatic strictures	▸ Diagnostic pancreatography ▸ Pancreatic cytology
Grade 2	▸ Stone extraction >10 mm ▸ Dilatation of stenosis/stent placement/nasobiliary drain in hilar tumours or benign intrahepatic strictures	▸ Cannulation of papilla minor
Grade 3	▸ BII anatomy ▸ Intrahepatic stone extraction ▸ Stone extraction with lithotrypsy	▸ Therapeutic pancreatic procedures, including pseudocyst drainage

A questionnaire with questions on the endoscopists’ experience with the form, why some forms were not completed and suggestions for improvement was filled out by all participating endoscopists to gain insight into their opinion on the added value of this quality measurement programme.

Statistical analysis

Data were analysed using the statistical software package PASW V.17. Descriptive statistics and graphs were used to characterise the data. χ² tests were used in order to test for differences between two groups. Regarding all results, a two-sided p-value <0.05 was considered statistically significant. Performance data on cannulation rate are expressed as group medians plus ranges unless stated otherwise.

Results

From January 2008 to December 2011, the five qualified endoscopists performing ERCP in our department participated in this study. A total number of 1691 ERCPs was performed by these endoscopists in this period. Fifteen hundred fifteen procedures were appraised using RAF-E. Adherence to completion of the self-assessment forms was 89.6%. All 176 non-assessed ERCPs were checked manually by deriving reports from our electronic endoscopy reporting system that captures all procedures. The CBD cannulation success rate of non-assessed procedures was not significantly different from self-assessed ERCPs (95.3% vs 94.1%; χ²=0.774, p=0.379).

The appraised procedures were further analysed. The average number of ERCPs per endoscopist per year was 116 (range 56–184). Table 2 presents an overview of indications and complexity for ERCP in this hospital. The distribution of difficulty degrees is shown in table 3. Figure 2 shows the ERCP difficulty degree distribution per endoscopist. Table 4 shows the results of performance data of all endoscopists who participated in this study (total procedural outcomes as well as separate per difficulty degree).

Table 2.

Overview of indications for ERCP

Indication	Number of procedures (total n=1515) (%)
Stones	346 (22.8)
Malignant stenosis	359 (23.7)
Benign stenosis	272 (18.0)
Chronic pancreatitis	190 (12.5)
PSC	90 (5.9)
Biliary leakage or trauma	79 (5.2)
Endoprosthesis change	41 (2.7)
Other	138 (9.1)

ERCP, endoscopic retrograde cholangiopancreatography; PSC, primary sclerosing cholangitis.

Table 3.

Distribution of difficulty degrees

Difficulty degree	n (%)
1	910 (60.1)
2	230 (15.2)
3	375 (24.8)

Figure 2.

Endoscopic retrograde cholangiopancreatography (ERCP) difficulty degree distribution per endoscopist.

Table 4.

Performance data

ERCP difficulty	Intent	Papilla (n)	Success (%)	Partial success* (%)	Failure (%)
All	CBD cannulation	All (1317)	94.1 (84.2–97.2)	0.6 (0–1.8)	5.6 (2.3–15.5)
		Virgin (602)	87.0 (79.2–95.5)	1.4 (0–4.0)	10.7 (3.2–20.2)
		Sphincterotomy (713)	98.4 (90.5–100)	0	1.6 (0–9.5)
		Previous failure (266)	86.7 (65.4–96.2)	0 (0–3.4)	13.3 (3.2–34.6)
		No previous failure (733)	96.2 (89.8–99.2)	0 (0–1.0)	3.6 (0.8–10.2)
	Sphincterotomy	n=377	100 (98.2–100)	0	0 (0–1.8)
	Stenting	n=808	97.8 (90.2–98.5)	1.0 (0–4.6)	2.1 (0–5.2)
	Stone extraction	n=296	86.8 (74.1–89.6)	6.6 (0–14.8)	8.3 (6.3–11.1)
1	CBD cannulation	n=892	91.9 (85.0–96.0)	0.7 (0–1.8)	7.0 (3.2–14.7)
	Sphincterotomy	n=254	100	0	0
	Stenting	n=498	97.4 (91.2–100)	0 (0–5.4)	2.6 (0–4.9)
	Stone extraction	n=201	91.3 (83.9–100)	0 (0–8.9)	5.7 (0–13.3)
2	CBD cannulation	n=229	100 (90.0–100)	0 (0–3.2)	0 (0–10.0)
	Sphincterotomy	n=46	100	0	0
	Stenting	n=163	97.0 (90.0–100)	0 (0–4.8)	0 (0–10.0)
	Stone extraction	n=53	75.0 (50.0–100)	16.7 (0–37.5)	0 (0–13.3)
3	CBD cannulation	n=196	85.7 (62.5–96.7)	0 (0–0.8)	14.3 (2.5–37.5)
	PD cannulation	n=266	85.7 (65.7–91.7)	0 (0–2.7)	14.3 (8.1–34.3)
	Spincterotomy	n=76	100 (83.3–100)	0	0 (0–16.7)
	Stenting	n=147	95.7 (81.3–100)	0 (0–4.5)	1.3 (0–18.8)
	Stone extraction	n=42	75.0 (50.0–100)	12.5 (0–25.0)	4.5 (0–25.0)

*Partial success regarding cannulation was defined as passage of contrast fluid or successful cannulation with a guidewire, but cannulation with the catheter was not achieved.

CBD, common bile duct; ERCP, endoscopic retrograde cholangiopancreatography; PD, pancreatic duct.

Indications per difficulty level

The most frequent indications in level 1 ERCPs were CBD stones (n=251; 27.6%), benign strictures (n=216; 23.7%) or malignant strictures (n=240; 26.4%). Malignant stricture was in the level 2 group, the most frequent indication for ERCP; this accounted for 41.7% of the total number (n=96). The major part of level 3 ERCPs was performed for pancreatic indications, such as endoscopic therapy in chronic pancreatitis (47.2% of all level 3 ERCPs).

Success of procedures

Procedural success was calculated for the most common indications: stone extraction and benign or malignant strictures. In 148 patients with stones, both sphincterotomy and stone extraction were attempted after cannulation of CBD. In these cases, sphincterotomy was successfully performed in all cases as well and complete stone extraction was successful in 92.3%. Procedural success in cases in which ERCP was indicated for suspicion of CBD stones was therefore 92.3% (range 84.2–94.4%). In patients with strictures (either benign or malignant; n=482), CBD cannulation success rate was 98.4% and stent placement was successful in 96.8% of the cases, resulting in an overall 95.3% (range 82.4–100%) procedural success rate.

The CBD cannulation success rate in patients with a virgin papilla compared with patients who had undergone a previous sphincterotomy was significantly different (87.0 (range 79.2–95.5) vs 98.4% (range 90.5–100); χ²=36.66, p<0.01). Analysing CBD cannulation success rate in previous ERCP failure versus no failed procedure before (86.7 vs 96.2%) showed a significant difference as well (χ²=13.88, p<0.01).

Questionnaire

The participating endoscopists were asked to give their opinion about the self-assessment programme through a short questionnaire. The common opinion was that this programme was valuable and useful to gain insight into performance. The form was found to be easy to fill out with clear questions. It took the participating endoscopists approximately 1 min to fill out the form. The endoscopists were also asked for reasons why some procedures were not assessed. The main reason for this was that they forgot to fill out RAF-E due to time pressure or busy programmes. Receiving feedback on performance was stated as important in order to achieve and maintain good adherence.

Discussion

In this study, we prospectively evaluated the performance of endoscopists with respect to ERCPs in a tertiary referral hospital using a self-assessment method. With a simple form, we were able to present a descriptive analysis of indications, difficulty degrees, cannulation success rates and therapeutic success. This quality assessment of 1515 procedures gave insight into performance of individual endoscopists as well as group performance. The self-assessment programme seems a reliable method to monitor quality and performance. With the rising attention for quality assurance and the expectancy that healthcare inspection will take measures within the near future in order to assess and assure endoscopic quality, this method of self-assessment seems a useful instrument to register performance. In addition to complication registration, which is already compulsory for all endoscopy departments in our country, this procedural registration will address a different and valuable aspect of procedural quality.

Development of the form

The demand for quality assessment in endoscopic procedures is increasing, but up until now, there is still no gold standard to assess the quality of ERCPs. Dutch guidelines state that an endoscopist is certified for ERCP when he has performed 100 procedures (Dutch Association of Gastroenterologists). Obviously, there is no scientific basis for this threshold number and the quality of these procedures remains unknown. ASGE guidelines state that a CBD cannulation success rate of ≥90% is an overall appropriate target, including experts, and that most endoscopists should be able to achieve a success rate of 85%.⁴ In general, gastroenterologists in smaller regional hospitals usually perform ERCPs with a level 1 difficulty degree. Technically more challenging procedures with difficulty level 2 or 3 are more often performed in tertiary referral centres. Questions regarding quality are being raised, such as the number of successful CBD cannulations and whether there was a difference in success between virgin papillas and patients with previous sphincterotomy. Numbers on the most frequently performed types of procedures are interesting as well. Another matter might be about a case with successful CBD cannulation, but failed stone extraction. On top of these questions, we wanted to encourage critically reflecting on one's performance. Our aim was to develop a self-assessment form that was easy to fill out and addressed the problems raised. A major drawback of too extensive evaluation forms is the inversely correlated drop in adherence to filling them out as we experienced in a pilot study. As reported by the participants, the most common reason not to fill out the form was lack of time during a busy programme. It was thus necessary to find a balance between the number of questions asked in the form and ease of completing it. Unfortunately, this implies an inevitable trade-off for a number of parameters that would have been interesting to monitor as well. In order to achieve compliance as high as possible and to develop a practical assessment tool for future, the most important outcome parameters were determined through expert opinion and this process resulted in the development of the RAF-E form used in this study. The final version of the form combines registration of procedural intention and outcome; we have used parts of previously validated assessment tools and metrics as described by Peter Cotton.^{17 18} Parameters that were scored as ‘partially successful’ were considered to be failures in the analysis in order to avoid any discussions on definitions of partial success. The results have shown that RAF-E is an easy-to-use device that provides insight into performance of individual endoscopists as well as larger groups. Unfortunately, the improvement plan was sparsely filled out by the experts, so it is difficult to make a statement on whether performance has improved afterwards. However, it might be worth considering linking RAF-E to the electronic endoscopy reporting system. This is a desirable step to take within the near future, which ensures the reliability of this method and provides easy accessible data for analysis. Since too much registration seems to be a burden for doctors in general, as well as for endoscopists, we believe that this linking of systems will result in even more reliable data.

What is known in the literature

Previous studies on ERCP quality focused mainly on complications, an accepted outcome parameter to assess quality.^19–21 However, next to complication registration, we believe that the procedural quality is of equal importance as stated above. Some study groups have published their performance data on, that is, cannulation success with numbers similar to our centre (92% to 94%).^{6 7 22} Since performing ERCP only for diagnostic purposes, for example, in suspected CBD stones^{23 24} is considered obsolete and risky, less invasive methods such as magnetic resonance cholangiopancreatography (MRCP) or endoscopic ultrasound (EUS) are recommended for diagnostics. ERCP should thus be limited to therapeutic interventions. Taking this into account, therapeutic or procedural success is an important outcome parameter as well. ASGE states that clearance of CBD stones should be achieved in >85% of the cases. In our centre the overall success rate was 86.8%, which is adequate. Stent placement was successful in 97.8% of patients, which is in line with the ASGE recommendations as well (>90% success rate). These data show that our RAF-E provides insight into performance criteria such as those formulated by ASGE. We used a modified version of the Schutz's classification in this study. An ASGE working party has proposed a revised complexity score with four difficulty categories. This score was not yet available at the start of this study.²⁵ The results of our study are in line with the findings of the ERCP Quality Network,²⁶ which is an electronic system where endoscopists can enter their data anonymously. As a result of this anonymity, no data verification can be performed and data are thus subject to bias. Moreover, one cannot recollect whether the ERCPs entered in the database are the only procedures performed by the participating endoscopists (registration bias). On the other hand, since the ERCP Quality Network is anonymous, there is less incentive to leave out failed cases. Our study was single centre; the ERCP Quality Network enables endoscopists from different centres to enter their procedures in the database.

Strengths and limitations

This is the first study to show the rationale for using a self-assessment programme in order to assess the quality of ERCPs and the prospective design is one of the strengths. In the Netherlands, no methods are available to gain insight into quality and performance of ERCP apart from retrospective database research. This results in the unique character of our data.

There are some limitations to this study. One possible source of bias is registration bias. The endoscopists themselves are responsible for filling out the assessment forms correctly. One might state that the forms are sensitive for falsification since they are not linked to an endoscopy report database. An endoscopist could just leave out the procedures that failed. Therefore, reports of all ERCPs performed in the time frame were extracted to quantify and evaluate non-assessed procedures. These non-assessed procedures were checked manually on procedural outcomes such as CBD cannulation success. There were no significant differences in outcome between assessed and non-assessed procedures, which makes it unlikely that procedures were left out on purpose. Another limitation of this study is that it was performed in a single tertiary referral centre. The performance numbers cannot be extrapolated to general endoscopists in smaller hospitals with a different workload and case mix. Moreover, there were quite some variations in numbers and case mix between the endoscopists in this study alone. This might have had some influence on the results, but this is a reflection of the workload and caseload in our endoscopy unit and probably many other endoscopy units across the world. However, a clear relation between numbers of ERCPs performed and outcome has not been established yet.²⁷ The numbers in this study are too small for a clear point of view on this topic, but when looking closer at our analysis, there was no correlation between volume and performance of the participating endoscopists. The one with the lowest volume was not the worst performer and vice versa. Finally, the impact of trainees on procedural outcome or success was not established in this study. Since this study was performed in a teaching hospital, trainees were involved in most ERCPs. It would be interesting to analyse their contribution in a new prospective study.

Implications

This study shows that a simple self-assessment form is a successful device to provide insight into quality of ERCPs on an individual basis as well as group performance. However, we experienced that in order to achieve and maintain a good adherence, reporting data to the participants on an individual basis was eminent, but this required time and dedication as well as one of the endoscopists stimulating the others to fill out the RAF-E forms. The next step is to roll out this self-assessment programme nationwide in the Netherlands, which will provide insight into quality and performance regarding ERCPs across the country. Next to investigating quality of ERCPs performed by experienced endoscopists, including trainees, in this self-assessment programme will provide additional information on learning curves on top of quality assessment.

What is already known on this subject.

• Although quality monitoring of ERCPs is extremely important, useful assessment tools are lacking.

What this study adds.

• Self-assessment is a valuable method to gain insight in ERCP performance.

• The RAF-E captures the most important procedural quality parameters and is easy to incorporate in daily practice.