Learning curves for three specific procedures by anesthesiology residents using the learning curve cumulative sum (LC-CUSUM) test

Abstract

Background

The learning curve cumulative sum (LC-CUSUM) test is an innovative tool that allows quantitative monitoring of individual medical performance during the learning process by determining when a predefined acceptable level of performance is reached. This study used the LC-CUSUM test to monitor the learning process and failure rate of anesthesia residents training for specific subspecialty anesthesia procedures.

Methods

The study included 490 tracheal punctures (TP) for jet ventilation, 340 thoracic epidural analgesia (TEA) procedures, and 246 fiberoptic nasal intubations (FONI) performed by 18 residents during their single 6-month rotation.

Results

Overall, 27 (14–52), 19 (5–41), and 14 (6–33) TP, TEA, and FONI procedures were performed, respectively, by each resident. In total, 2 of 18 residents achieved an acceptable failure rate for TEA according to the literature and 4 of 18 achieved an acceptable failure rate for FONI, while none of the residents attained an acceptable rate for TP.

Conclusions

A single 6-month rotation in a reference teaching center may not be sufficient to train residents to perform specific or sub-specialty procedures as required. A regional learning network may be useful. More patient-based data are necessary to conduct a risk adjustment analysis for such specific procedures.

Introduction

As assessments of healthcare quality become more important in medical practice [1], statistical process-control methods initially developed to monitor the quality of manufactured goods are being increasingly used to monitor clinical performance [2]. The cumulative sum (CUSUM) is one of the methods used in the medical field. The learning curve CUSUM (LC-CUSUM) test is an alternative to the CUSUM test that was developed to focus specifically on the learning period of a procedure [3,4]. Statistical process-control methods were first used to monitor surgical performance in pediatric cardiac surgery [5] and are still used to monitor cardiac surgeons' performance and their patients' outcomes [6]. Statistical process-control methods have been used in some studies to construct learning curves for anesthesiology procedures, such as peripheral vein cannulation [7], tracheal intubation and mask ventilation [7,8,9], epidural analgesia [8], transverse abdominal plane block [10], or ultrasound assessments for epidural analgesia [11,12].

The French anesthesia residency program has certain mandatory rotations, such as pediatric and obstetric anesthesia, but there is no obligation for the subspecialty rotation. Other anesthesia training programs offer sub-specialty rotations scheduled for managing specific procedures [13]. However, technical competency for specific procedures in France improves only during an elective subspecialty rotation.

Oncologic abdominal and cervicofacial surgery comprises a large part of our activities, and our department is considered a reference center for fiberoptic nasotracheal intubation (FONI) (> 250 procedures/year), thoracic epidural analgesia (TEA), and tracheal puncture (TP) for high-frequency jet ventilation (HFJV) (> 500 procedures/year) by French academics and residents. We have been searching for an objective and individual performance measurement method to assess the learning process for these procedures correctly. Thus, this study assessed the learning curve of residents performing FONI, TEA, and TP procedures during a 6-month rotation using a dedicated statistical method (LC-CUSUM test).

Materials and Methods

During three consecutive rotations, data from 18 residents (six per rotation) were collected for all performed procedures. After local ethics committee agreement to waive informed consent, the residents were recruited for the LC-CUSUM test evaluation at the beginning of their 6-month rotation in our department. Our current practices remained unchanged.

Protocol and measurements

This study focused on TP, TEA, and FONI. As the participants had already completed most of their residency program, they had already completed the obstetrics rotation and had practiced more than 150 lumbar epidural analgesia procedures. Despite this experience, all residents had performed fewer than 10 TEA procedures before our departmental rotation. They were new to TP for HFJV and most were new to FONI (four FONI procedures for the most experienced resident). The residents were informed about the assessment and the criteria for failure and success of each procedure in our department on the first day of the rotation. The residents received formal teaching about each of the procedures during the first week of the rotation and assisted a staff practitioner with one procedure. Then, the first procedure was performed under active supervision and residents were advised appropriately. The first attempt at each of the three procedures was considered a rehearsal, and no data were collected for the final analysis. Data were collected in a personal paper log book, in which the failure criteria were recorded.

Anesthesia was induced prior to HFJV using a total intravenous anesthesia technique with remifentanil and propofol. Then, the trachea was punctured through the cricothyroid membrane with an arterial Leader Cath 14-gauge catheter (Vygon, Ecouen, France) using the Seldinger method. The criteria for failure were two punctures or more and catheter misplacement after checking during direct hypolaryngoscopy with or without a complication (intraoperative subcutaneous emphysema or hypoxia with SpO₂ < 90% were considered) [14]. TEA was performed by locating the epidural space using a blind loss of resistance technique to saline with an 18-gauge Tuohy needle (Portex Epidural minipack; Smiths Medical, Ashford, UK). The catheter was inserted 5 cm into the epidural space. A test dose was administered before anesthesia was induced, and epidural analgesia was used during surgery following the departmental analgesic protocol [15]. The procedure was considered a failure if epidural analgesia was insufficient in the postoperative care unit, if more than two punctures were needed, if the resident was unable to insert the catheter, or if the dura mater was perforated when the catheter was inserted. Fluoroscopy and ultrasound were not used to help guide the needle tip, as all of our “last year” residents had already completed the obstetrics rotation and were proficient in obstetric anesthesia, including epidural analgesia, and were fully aware of the complications related to epidural puncture. FONI was realized under sedation with a remifentanil target control infusion, which is our standard practice. Endoscopy was performed with a video feedback system in all cases to allow confirmation of placement under supervision by a senior practitioner. The procedure was considered a failure if the resident was unable to perform the entire procedure without any intervention from a tutor, if sedation was inadequate (patient expressed unacceptable pain with a numerical verbal scale score > 3, movement, or agitation), or if a complication occurred (hypoxia with SpO₂ < 90%, bleeding in airway, or bronchospasm) [16].

All procedures were considered successful if no failure criteria were met.

Data analysis

The LC-CUSUM test was developed to determine when a trainee has reached a predefined level of performance [3]. The LC-CUSUM sequentially tests the null hypothesis that “performance is inadequate” against the alternative that “performance is adequate.” The LC-CUSUM test computes the “St” score from successive outcomes; a success increases the score and a failure decreases it. Numerically, St = max (St0 − 1 + Wt), whereas St0 = 0 and Wt = log[(1 − p0)/(1 − p0 − Δ)], if the procedure is a success, and Wt = log[p0/(p0 + Δ)], if the procedure is a failure. p0 is the proportion of failure under an adequate performance level, Δ is an acceptable deviation from adequate performance, and t is the number of attempts and is > 0. Once the score reached the predefined limit “h,” the test rejects the null hypothesis in favor of the alternative, and performance is deemed adequate.

The LC-CUSUM score was plotted along the y-axis against successive procedures along the x-axis. As long as the score remained in the continuation region, namely, between the x-axis and the decision limit, “h,” performance was not considered adequate and monitoring was continued. The score increases as the number of failures decreases until it crosses the limit “h” where proficiency is declared. A particular feature of the LC-CUSUM test is that it incorporates a holding barrier at zero that cannot be crossed. Therefore, the “St” score remains zero if the trainee accumulates numerous successive failures and, consequently, the LC-CUSUM remains responsive to a decrease in the number of failures at all times. For example, if a failure occurs due to a technical misunderstanding, the trainee does not have to compensate unnecessarily for all accumulated failures, indeed once he overcomes the problem and may capitalize on successes only.

Based on a consensus from published data [14,15,16], acceptable failure rates (defined proficiency) were 1, 10, and 18% for the TP, TEA, and FONI, respectively; deviations from these levels that were considered acceptable were 1, 5, and 9%, and the defined equivalence zones were 2, 15, and 27%, respectively. Computer simulations were performed to obtain limits and their corresponding true and false discovery rates. Tables 1, 2, 3 show the TP, TEA, and FONI computer simulations. The true discovery rate (TDR) is the proportion of alarms emitted under the acceptable failure rate (alternative hypothesis), and the false discovery rate (FDR) is the proportion of alarms emitted under the non-acceptable failure rate; both rates are defined for a given number of procedures. A high TDR and low FDR are the objectives, but these rates vary together; thus, there is a necessary trade-off between a test that is sensitive and a test that will yield few false alarms. The decision limits were chosen to provide a TDR of 80%, regardless of the FDR; performance on the tests was calculated arbitrarily for 50 of each of the three procedures, as that is the maximum number of procedures that could be performed by each resident according to our database. The number of procedures during which the test is run is also critical to test performance. Namely, the more procedures a trainee is given, the more likely the test will be able to separate the null from alternative hypotheses. For example, if the trainee has a true performance of 10% for TEA (adequate level) and the TDR is 80%, there is an 80% chance of the test signaling (indicating acceptable failure rate) more than 50 procedures. If the FDR is 15%, true performance is 20% (inadequate level), and there is a 15% chance of emitting a false alarm from more than 50 procedures.

Table 1. True Discovery Rate according to the Proportion of Failure under an Adequate Training Level (P0) for Tracheal Puncture.

h	p1%	p1.5%	p2%	p2.5%	p3%
0.025	100	100	100	100	100
0.050	100	100	100	100	100
0.075	100	100	100	100	100
0.100	100	100	100	100	100
0.125	100	99.98	99.91	99.87	99.81
0.150	99.98	99.93	99.86	99.56	99.19
0.175	99.81	99.54	98.77	98.03	96.64
0.200	99.39	98.60	97.35	95.57	94.00
0.225	98.15	96.55	94.09	91.06	87.17
0.250	97.41	94.91	90.96	87.02	82.01
0.275	92.73	88.47	83.06	76.96	72.05
0.300	89.99	82.92	77.58	71.62	64.26
0.325	84.82	76.90	95.14	58.26	51.50
0.350	81.50	72.99	65.14	58.06	51.50
0.375	78.29	69.32	60.11	52.86	46.46
0.400	74.76	64.25	55.00	46.01	38.60
0.425	71.69	59.83	51.41	41.82	35.62
0.450	66.22	55.97	45.23	36.88	30.11
0.475	65.21	51.60	42.79	34.24	26.80
0.500	60.91	47.88	37.11	29.02	22.89
0.525	0	0	0	0	0
0.550	0	0	0	0	0
0.575	0	0	0	0	0
0.600	0	0	0	0	0
0.625	0	0	0	0	0
0.650	0	0	0	0	0
0.675	0	0	0	0	0
0.700	0	0	0	0	0
0.725	0	0	0	0	0
0.750	0	0	0	0	0
0.775	0	0	0	0	0
0.800	0	0	0	0	0

Fifty procedures were simulated. Hour is the decision limit, and p(x)% is the performance rate for the procedure.

Table 2. True Discovery Rate according to the Proportion of Failure under an Adequate Training Level (P0) for Thoracic Epidural Analgesia.

h	p10%	p12.5%	p15%	p17.5%	p20%
0.50	99.29	97.54	94.25	89.58	83.33
0.55	98.30	94.97	90.37	82.32	74.00
0.60	96.55	91.80	84.55	74.27	63.63
0.65	94.23	87.39	77.98	66.89	54.97
0.70	91.64	82.18	70.85	58.45	46.08
0.75	88.31	77.27	63.43	51.24	38.48
0.80	87.30	74.89	61.96	49.39	37.47
0.85	83.71	70.45	56.35	42.93	31.54
0.90	80.30	65.37	51.97	37.34	26.11
0.95	77.96	61.13	45.39	31.09	21.55
1.00	72.57	56.09	40.27	26.33	17.13
1.05	68.10	50.72	35.41	22.73	13.94
1.10	63.39	45.49	30.72	19.60	11.39
1.15	56.62	41.26	25.66	16.44	9.50
1.20	55.75	38.33	25.62	14.93	8.64
1.25	53.77	35.51	22.30	13.27	7.39
1.30	50.23	32.45	20.26	10.71	6.26
1.35	47.00	29.56	17.65	9.46	4.52
1.40	43.24	25.63	14.84	8.07	3.66
1.45	39.81	23.51	12.76	6.17	2.98
1.50	37.17	20.29	10.64	4.99	2.55
1.55	30.21	17.08	8.91	3.90	1.46
1.60	28.44	15.60	8.41	3.31	1.47
1.65	27.26	14.13	7.05	3.50	1.21

Fifty procedures were simulated. Hour is the decision limit, and p(x)% is the performance rate for the procedure.

Table 3. True Discovery Rate according to the Proportion of Failure under an Adequate Training Level (P0) for Fiberoptic Nasal Intubations.

h	p18%	p22.5%	p27%	p31%	p36%
0.5	99.99	99.75	99.16	96.86	92.49
0.6	99.75	98.86	96.17	89.89	79.73
0.7	99.35	96.93	90.60	78.27	63.79
0.8	98.97	95.31	87.70	75.84	60.91
0.9	97.44	90.83	77.93	61.97	42.97
1.0	94.84	83.84	67.88	47.66	31.58
1.1	92.62	80.22	59.72	40.20	23.70
1.2	88.85	71.25	50.57	30.91	17.22
1.3	84.47	64.34	42.99	23.42	11.98
1.4	80.64	58.83	35.52	18.26	8.68
1.5	77.69	54.93	30.97	16.20	7.17
1.6	72.47	48.11	26.06	12.47	4.68
1.7	67.04	40.74	20.67	9.15	3.47
1.8	63.31	37.97	18.51	7.26	2.61
1.9	58.26	32.22	14.49	5.88	1.78
2.0	53.86	28.01	11.52	4.22	1.36
2.1	50.25	24.29	9.70	3.05	0.96
2.2	45.95	22.09	8.38	2.65	0.63
2.3	40.61	18.42	6.24	2.28	0.54
2.4	36.74	15.68	4.73	1.51	0.38
2.5	34.46	14.13	4.03	1.17	0.29
2.6	30.20	10.83	3.23	0.85	0.19
2.7	26.98	9.50	2.48	0.51	0.08
2.8	21.79	7.35	1.96	0.47	0.08
2.9	20.85	6.52	1.68	0.30	0.07
3.0	17.86	5.41	1.12	0.18	0.04

Fifty procedures were simulated. Hour is the decision limit, and p(x)% is the performance rate for the procedure.

The values for an 80% TDR were 0.9 for TEA, 0.375 for TP, and 1.4 for FONI.

Results

Eighteen residents performed a total of 1,049 procedures during their rotation.

Tracheal puncture

The LC-CUSUM results for TP are presented in Fig. 1. A total of 490 TP procedures were performed. Residents performed a median of 27 procedures (interquartile range [IQR], 21–31). In total, 451 (92%) of the procedures were successful regarding the chosen criteria. Among failures, 25 (5%) were misplacements of the catheter, 11 (2%) resulted in subcutaneous intraoperative emphysema, and 5 (1%) resulted in hypoxic episodes. Two patients had concomitant emphysema and misplacement. The decision limit was never crossed; thus, no trainee was declared proficient with TP at the end of the rotation.

Fig. 1. Learning curve cumulative sum (LC-CUSUM) test results for tracheal puncture for the 18 residents. Dashed line represents the decision limit “h” at 0.375. No residents crossed this decision limit.

Thoracic epidural analgesia

The LC-CUSUM results for TEA are presented in Fig. 2. A total of 340 procedures were performed (median, 19/resident; IQR, 14–23). According to the criteria, 285 (83%) procedures were successful and 27 were failures (8%) (with resident unable to insert the catheter, in 13 cases (4%) more than two attempts, 12 cases (4%) of inefficient analgesia, and three (1%) dura mater punctures). The decision limit was crossed by two (11%) residents after 17 and 34 procedures.

Fig. 2. Learning curve cumulative sum (LC-CUSUM) test results for thoracic epidural analgesia for the 18 residents. Dashed line represents the decision limit “h” at 0.9.

Fiberoptic nasotracheal intubation

The LC-CUSUM results for FONI are presented in Fig. 3. A total of 246 procedures were performed with 226 (92%) successes; two (0.7%) residents did not complete the entire procedure alone, one (0.3%) did not manage sedation adequately, and 17 (7%) complications occurred, mostly hypoxia. Residents handled a median of 14 (IQR, 9–16) intubations during their rotation. Two residents performed more than 22 procedures. The decision limit was crossed by four (22%) residents after 13 (n = 3) and 17 procedures.

Fig. 3. Learning curve cumulative sum (LC-CUSUM) test results for fiberoptic nasotracheal intubation for the 18 residents. Dashed line represents the decision limit “h” at 1.4.

Discussion

The LC-CUSUM results revealed that all but four residents did not manage to demonstrate competency during their rotation and were not considered competent with the procedures after their rotation in our department. Our results are not consistent with previously published studies. For example, de Oliveira Filho and Komatsu [7,9] reported that ≥ 50% of residents demonstrate competency for lumbar epidural analgesia or orotracheal intubation procedures. Some explanations of this discrepancy should be considered. The adequate and inadequate performance rates chosen may not have been appropriate. We chose published data [14,15,16] to set performance rates and compare results from studies in other departments. de Oliveira Filho [7] determined the acceptable performance rate for learning lumbar epidural analgesia arbitrarily from samples of procedures performed by instructors. They used 20 and 40% as acceptable and unacceptable failure rates, respectively, and 45% of the residents attained an acceptable failure rate with these limits. Kestin [8] used rates for the same procedure taken from the consensus among staff anesthesiologists in their department (5 and 10% for acceptable and unacceptable failure rates, respectively). They reported that 41% of residents exceeded the chosen failure rate and concluded that the failure rate might be too stringent. They reported that 39–67 procedures were needed to achieve competency in lumbar epidural analgesia, according to the CUSUM test. No information on adequate performance rates for TP and FONI is available in the literature.

The choice of TDR and FDR plays an important part in monitoring efficiency, as our simulations showed. Test performance depends on the difference between the adequate performance level and the acceptable deviation from that level and on the number of procedures during which the trainee is observed. The larger the difference and the longer the observation period, the better the test performance. Our simulations showed that trainees needed to perform approximately 65 FONI (h = 1.67), 140 TEA (h = 1.82), and 520 TP procedures to obtain a TDR of 80% with an FDR of 5%, where the difference between performance under the null and alternative hypotheses is extremely low (1%) (h = 1.88). If a trainee is denied the appropriate number of procedures, then we are at risk of not detecting proficiency in due time by setting the limit “h” too high or we are at risk of having too many false alarms by setting the limit “h” too low. Instructors could decide not to wait for the required number of procedures when assessing competency. In practice, the test properties are mediocre due to the very stringent level of performance required for some procedures, such as TP. Therefore, a dilemma arises in which the test properties are improved, the trainee is allowed to complete a relatively large number of training procedures, or the number of procedures is limited but the properties of the test remain mediocre. Only four residents achieved adequate FONI performance. Using an efficient test to monitor learning, such as the LC-CUSUM, for a 6-month rotation in a reference center may be insufficient to consider that a resident has reached proficiency. This could be an effective approach if a regular collection is paired with an attentive survey from residency heads at the beginning of the residency. However, completing and interpreting charts could be difficult due to discontinuous activity [8,17] and the effectiveness of such an undertaking has not been evaluated. Another possibility is that the residency is organized differently, so proficiency might not be achieved but performance is acceptable. Rotations could be made to fulfill specific goals rather than fulfill a predefined time period in a department [13]. Residents would have to continue training as long as these goals were not met. An efficient statistical process-control method is mandatory for such an approach to prevent excessive training time on a specific topic [13].

Few studies have investigated teaching specific procedures for airway management. Teaching HFJV is difficult, as this procedure is not in general use, despite its validation as part of airway management algorithms [18]. Although the rate of complications is directly related to the number of procedures performed [19], minimal skill during residency has not been claimed from a simulator setting [13]. Cricothyroidotomy has only been studied on mannequins or cadavers [20,21]. Most FONI studies included either sleeping patients with no difficult airways or mannequins [22,23], and the time to complete these procedures was the main outcome studied, rather than efficacy or safety. Furthermore, such conditions for carrying out the procedures have nothing to do with patients undergoing head and neck surgery. Real-life training seems unavoidable until an acceptable level of training is reached for technical and non-technical skills during complex procedures, such as TEA or FONI, as the use of mannequins is considered inappropriate [23,24].

Using a risk adjustment for a procedure would be of great benefit. Unlike some authors [9,25], we did not use a risk-adjusted test, mainly because we were unable to find external data to quantify patient risk level, except for epidural analgesia [26,27]. This was a limitation of our study and is consistent with the requirement for specific data, as reported by Komatsu et al. [9]. We have started collecting demographic data in a new clinical database to fulfill this requirement concerning airway management.

In conclusion, the LC-CUSUM test enabled learning of unusual and specific procedures by residents to be assessed. Reference centers performing subspecialty procedures may not provide sufficient training for these procedures during a single 6-month rotation. An arbitrary number of attempts or training time may not adequately respond to individual training variations, so consideration should be given to monitoring the learning process from the beginning or redefining the residency curriculum.