Endobronchial Ultrasound-Transbronchial Needle Aspiration of Mediastinal and Hilar Lymphadenopathy Learning Curve

Endobronchial ultrasound (EBUS)-Transbronchial needle aspiration (TBNA) has gained popularity for diagnosis as well as staging patients with suspected lung cancer. It is a minimally invasive procedure that allows real-time sampling of mediastinal and hilar lymphadenopathy. It has a comparable diagnostic yield to the gold standard procedure, mediastinoscopy, in histological evaluation of lymphadenopathy as well as in staging of thoracic malignancies.¹ However, the high accuracy of EBUS-TBNA was attained by physicians who are considered experts in the field and it will be very difficult to replicate the yield with inadequate training. This is important, since suboptimal knowledge about mediastinal anatomy, EBUS images and the technique itself will lead not only to procedure related complications but to inaccurate lung cancer staging as well.

The American Thoracic Society, European Respiratory Society and the American College of Chest Physicians recommend 40 to 50 procedures to acquire competency.^2,3 These recommendations were made before the introduction of the real-time EBUS linear-probe bronchoscope and are still a matter of debate. It is also well recognized that many medical centers are not following these guidelines. How many supervised procedures are needed to say that a physician has attained adequate competency in this procedure is still an unsettled issue.^4,5

The cumulative sum (cusum) analysis is a method used to continuously monitor performance against an established standard to attain an acceptable failure rates in the procedure performed.⁶ It has been used to construct learning curves in different medical specialties tracking trainees’ performance.^7,8 We applied cusum analysis to the learning of EBUS-TBNA in evaluating enlarged hilar and mediastinal lymphadenopathy to an operator who was not previously trained in the procedure and without any supervisor intervention. The diagnostic accuracy was also reported.

Methods and materials

Patients

This was a retrospective chart review approved by the Institutional Review Board of Tulane University and Southeast Louisiana Veterans administration with waiver of the informed consent. One hundred and fifty consecutive patients with enlarged lymph nodes (> 1 cm on short axis) on chest-computed tomography who underwent EBUS-TBNA were enrolled in the study. EBUS-TBNA was performed in patients meeting the following criteria: (1) enlarged mediastinal/hilar LN (≥1 cm) revealed by chest CT, (2) informed consent for EBUS-TBNA examination, and (3) no contraindication to the procedure. Among the150 patients included in the study, were 127 men and 23 were women. The mean age was 60.5 years (29–89 years).

Endobronchial Ultrasound Transbronchial needle Aspiration

A 7.5-MHz convex linear array ultrasonic transducer encased by a water-inflatable balloon at the distal tip (Olympus America; Center Valley, PA) was used for all patients. The bronchoscope was equipped with Doppler function for the identification of blood vessels. The EBUS scope was inserted through the endotracheal tube or LMA after induction of general anesthesia. Lymph nodes were identified in real time using the Olympus ultrasound processor (EU-C60), and the balloon was inflated to maintain adequate contact with the bronchial wall and to allow better visualization when necessary. A dedicated 21 and 22-gauge needle (NA-201SX–4022; Olympus Ltd.) was used for aspiration from targeted lymph nodes under real-time ultrasound guidance. Once the tip of the needle was confirmed as having reached the target lesion, the needle was moved back and forth while suction was applied. Three cytological evaluations were performed for each target lymph node. The procedure involved rapid on-site evaluation (ROSE) of specimens by a cytologist.

Statistical Analysis

Procedures were divided into six sequential groups of 25 each (1 to 25, 26 to 50, 51 to 75, 76 to 100,101 to 125 and 126 to 150). We defined a true positive (TP) result by the presence of any abnormal pathology (benign or malignant), a true negative (TN) result if the sample was normal, a false negative (FN) result if the sample was nondiagnostic or normal after completion of EBUS-TBNA but abnormal after a confirmatory procedure and a false positive (FP) if the sample was abnormal after completion of EBUS-TBNA but normal after a confirmatory procedure. Diagnostic accuracy was defined as (TP+TN)/(TP+TN+FP+FN). We evaluated trends of accuracy by using the Cochran–Armitage (C-A) trend test on one degree of freedom.

In order to construct a cusum analysis, the following variables should be defined (Table 1). The graph starts at cusum score of zero. For each successful attempt, S is subtracted from previous cusum score. However, 1-S is added to the previous cusum score if an attempt has failed. Therefore, a negative or stable trend on the graph indicates success but a positive trend indicates failure to achieve competence. H0 and H1 can be marked as horizontal lines on the graph. When the line crosses H1 from below, the actual failing rate is significantly greater that the acceptable failure rate with a probability of type I error. However, when the line crosses H0 from above, the true failure rate does not differ significantly from the acceptable failure rate with a probability of type II error. Using Kemp et al article⁹, we set P0, P1, α and β at 10%, 20%, 10%, 10% respectively. Thus, H0 andH1 were both calculated as 2.71.

Table 1.

Definitions and calculations for cusum analysis

P0	acceptable failure rate
P1	unacceptable failure rate
Type I error (probability that operator is competent)	α
Type II error(probability that operator is incompetent)	β
P	ln(P1/P0)
Q	ln [(1−P0)/(1−P1)]
S successful attempt	Q/(P+Q)
1-S failure attempt	1− [Q/(P+Q)]
a	ln ((1− β)/α)
b	ln ((1− α)/β)
H0 value between each acceptable decision level	b/(P+Q)
H1 value between each unacceptable decision level	a/(P+Q)

Results

The average procedural time was 48.5 minutes (range, 20–80 minutes). Two hundred and sixty two mediastinal/hilar lymph nodes were biopsied from the 150 patients. The average lymph node size was 1.9 cm (range, 1.0–3.5 cm) along the long axis and 1.4 cm (range, 1.0–2.4 cm) along the short axis. All patients with nondiagnostic results subsequently underwent mediastinoscopy for re-biopsy of the lymph nodes sampled by EBUS-TBNA.

Final diagnoses of all the 150 patients included in the study are listed on Table 2. Diagnostic accuracy (in percentage) curve is shown in figure 1. Dividing the number of procedures into sextiles, accuracy increased from 72% in the first, to 80% in the second and then 88% in all of last three sextiles. Improved diagnostic accuracy was not observed after the 75^th attempt (C-A trend test P value = 0.091).

Table 2.

Diagnoses of Patients

Diagnosis	Number
Lung cancer	97
Benign disease	31
Sarcoidosis	8
Extra pulmonary Cancer	8
Lymphoma	2

Figure 1.

Endobronchial ultrasound diagnostic accuracy curve

Learning curve using cusum chart is shown in figure 2. It is evident that the curve started to show negative deflection between the 55^th and 60^th procedure and did not show positive trend afterwards.

Figure 2.

Cumulative sum chart for endobronchial ultrasound. Line represents learning curve of the operator

Discussion

In this study we showed that at least 55 to 60 procedures were needed by an operator, without previous training or mentoring, to acquire sufficient level of competency in EBUS-TBNA.

Multiple studies have suggested that at least 60 to 100 procedures are needed to achieve an acceptable diagnostic yield.^10–12 However, it is very important to recognize that physicians vary in the rate they acquire skills during learning any procedure and a fixed number should not be set forth before acquiring the skills. Rather, a tool to evaluate the trainee level of expertise might be used and then a learning curve could be followed up to evaluate their skills.

Davoudi and colleagues¹⁰ created an objective assessment tool to master skills and knowledge of EBUS-TBNA. The endobronchial ultrasound skills and tasks assessment tool evaluated operators at different levels (beginners, intermediates and experienced) in airway introduction, navigation, ultrasound image acquisition, identification of mediastinal nodal and vascular structures, sampling and recognition of EBUS/computed tomography images of mediastinal structures. This tool was able to classify physicians according to the level of expertise. In addition, completion of more than 50 to 100 procedures contributed minimally to test scores.

In our study, the diagnostic accuracy increased by 16% from 72 % (procedures 1 to 25) to 88 % (procedures 51 to 75) during the first three sextiles. There was non-significant trend (C–A trend test P=0.091), since diagnostic accuracy remained 88% after the 75^th procedure. Multiple studies have shown that the diagnostic accuracy for EBUS-TBNA is between 84% and 98%.^13,14

Fernández-Villar et al¹¹ reported the diagnostic accuracy of two bronchoscopists undergoing EBUS-TBNA. One hundred and twenty patients were involved and each consecutive twenty cases were grouped together reaching total of six. Accuracy improved steeply from 70% to 89.9% (19.8% difference) between the first and the fourth group (i.e. procedure 61 to 80) and only 4.7% afterwards. In contrast to our operator, the two operators had several practical workshops and performed 5 to 10 supervised procedures before the study.

In a recent study, higher diagnostic yields were found in hospitals with higher volume of EBUS-TBNA.¹⁵ However, with the wide availability of EBUS in academic as well as community centers it is important to set forth a standard for physicians to achieve before performing the procedure without supervision. EBUS-TBNA should be used as a tool for both diagnosis and staging and inadequate knowledge will result in disastrous consequences such as undergoing curative surgical procedure when disease is not salvageable or vice versa.

It is time to move from setting a minimum number of procedures to certify trainees in pulmonary medicine into assessing each operator individually. Using a cusum analysis to construct a learning graph, as in our study and Kemp et al article⁹, might provide a simple tool to assess competency levels and probably allow mentors to intervene early if needed.

Our study was limited by a single untrained operator and his ability to achieve competency in EBUS-TBNA procedure after performing 55–60 consecutive procedures with acceptable accuracy. However, the number of procedures needed to achieve adequate competency was comparable to trainees in other studies.^11–12 In addition, Kemp et al⁹ showed that different operators attain adequate knowledge in EBUS-TBNA procedure at variable speed. Thus, we believe that gaining competence for untrained operator in the procedure require a learning curve rather than focusing on number of procedures done due to variable speed in attaining this knowledge among different trainees.

In conclusion, we would like to share our experience with centers where EBUS-TBNA is available for physicians to use. Having a tool, such as a learning curve, will allow operators to develop adequate competency level, each according to his/her own speed of acquiring knowledge and thus providing better care, safety, and treatment for patients.