Evaluation of Appropriate Mediastinal Staging among Endobronchial Ultrasound Bronchoscopists

Abstract

Rationale: Endobronchial ultrasound (EBUS) has transformed mediastinal staging in lung cancer. A systematic approach, beginning with lymph nodes contralateral to the primary tumor (N3), is considered superior to selective sampling of radiographically abnormal nodes. However, the extent to which this recommendation is followed in practice remains unknown.

Objectives: To assess the frequency with which pulmonologists, pulmonary fellows, and interventional pulmonologists endoscopically stage lung cancer appropriately.

Methods: Bronchoscopists currently performing EBUS were surveyed about their practice patterns, procedural volume, and self-confidence in EBUS skills; they then performed a proctored simulated staging EBUS. The primary outcome was the proportion of participants who appropriately initiated ultrasonographic evaluation with the N3 nodal stations in a simulated patient undergoing EBUS for mediastinal staging.

Results: Sixty physicians (22 interventional pulmonologists, 18 general pulmonologists, and 20 pulmonary fellows) participated in the study. The rates of appropriate staging by study group were 95.5% (21 of 22) for interventional pulmonologists, 44.4% (8 of 18) for general pulmonologists, and 30.0% (6 of 20) for pulmonary fellows (P < 0.001). Increased procedural volume correlated with appropriate staging practices (P < 0.001). Within each group, we assessed the concordance between self-confidence in EBUS and simulation performance. Among interventional pulmonologists, the concordance was 95.4%, followed by 61.1% for general pulmonologists and 40.0% for pulmonary fellows.

Conclusions: General pulmonologists and pulmonary fellows were less likely than interventional pulmonologists to perform appropriate EBUS staging. In addition, the lack of concordance between self-confidence and appropriate staging performance among noninterventionists signals a need for improved dissemination of guidelines for EBUS-guided mediastinal staging.

In patients with lung cancer without distant metastatic disease, assessment of mediastinal nodal involvement is critical in prognostication and therapeutic planning. Even in the absence of radiographically abnormal mediastinal lymph nodes, the prevalence of mediastinal (N2/N3) involvement consistently exceeds 10% in tumors with high-risk features such as central location or radiographic N1 disease (1, 2). Although cervical mediastinoscopy was traditionally considered the “gold standard” for preoperative mediastinal staging, the advent of linear probe endobronchial ultrasound (EBUS) dramatically changed the staging landscape.

The most recent iteration of the American College of Chest Physicians (ACCP) guidelines for staging of non–small cell lung cancer recommends needle techniques (EBUS or endoscopic ultrasound) as the first test of choice for invasive mediastinal staging (2). Current guidelines emphasize the need for surgical confirmation of lymph nodes with a high probability of malignancy after negative needle-based sampling. However, given the more recent literature, which indicates that EBUS is equivalent—if not superior—to mediastinoscopy, it is tempting to assume that a negative EBUS staging examination reduces the probability of nodal metastasis to the point that additional invasive staging is no longer necessary (3, 4). This assumption is likely to be erroneous if the intrathoracic lymph nodes are not systematically examined and sampled during the EBUS.

Adoption of EBUS bronchoscopy into clinical practice has occurred at a rapid pace; however, it is unclear how often an appropriate systematic examination is performed. Systematic EBUS staging involves endosonographic imaging of at least the 4R, 4L, and 7 lymph node stations and sampling of the largest lymph node more than 5 mm in short-axis diameter at each station, beginning at the N3 (contralateral) nodal stations, to avoid needle contamination (5, 6). When performed in this manner, EBUS–transbronchial needle aspiration has been shown to be sensitive in excluding mediastinal metastasis (3, 7–9). It should be noted that EBUS–transbronchial needle aspiration is also an excellent diagnostic tool for evaluating isolated mediastinal adenopathy or for obtaining molecular markers from a known malignancy; however, physicians must not confuse this focused approach with true EBUS-guided mediastinal staging (10).

What constitutes sufficient training to perform EBUS-guided mediastinal staging is undefined, and there are multiple “avenues” available to physicians planning to incorporate EBUS into clinical practice. Although interventional pulmonary fellowships provide the most intensive training in bronchoscopic procedures such as EBUS, only a small number of formal interventional pulmonary programs exist in the United States. Most general pulmonary fellowships now provide trainees with exposure to EBUS bronchoscopy, but the depth of exposure varies, with only 30% of fellowships reporting established protocols to evaluate EBUS competency (11). Practicing pulmonologists who want to incorporate EBUS into their clinical practice can attend commercial training courses to gain technical competency, but depending on institutional credentialing requirements, exposure can be as limited as attending a brief product in-service training session.

Our study evaluated how provider training correlates with appropriate staging practices among EBUS bronchoscopists. In addition, we examined whether self-confidence in EBUS skills was associated with appropriate staging practices. We hypothesized that general pulmonologists and pulmonary fellows currently performing EBUS in clinical practice are less likely than interventional pulmonologists to endoscopically stage lung cancer appropriately. Some of the results of this study have been previously reported in the form of an abstract (12).

Methods

This protocol was approved by the Institutional Review Board of the University of Texas M.D. Anderson Cancer Center (Protocol PA15-0836).

Subjects

This cross-sectional study compared three groups of bronchoscopists currently performing EBUS in clinical practice:

• •Interventional pulmonologists: Board-certified or board-eligible interventional pulmonologists. Physicians currently enrolled in interventional pulmonary fellowships were also included in this group.
• •General pulmonologists: Board-certified or board-eligible pulmonologists who are not board-certified or board-eligible interventional pulmonologists but perform at least 10 EBUS procedures per year.
• •Pulmonary fellows: General pulmonary fellows who have performed at least 10 EBUS bronchoscopies under supervision within fellowship training during the past year.

Participants self-identified as interventional pulmonologists, interventional pulmonary fellows, general pulmonologists, or general pulmonary fellows based on the preceding criteria. Interventional pulmonary fellows were grouped together with interventional pulmonologists for analysis. With the exception of those who self-identified as current interventional pulmonary fellows, there was no attempt made to verify duration or extent of training. Participants were recruited for participation while attending the ACCP 2015 Annual Meeting in Montreal, Canada (October 23–28, 2015). To recruit participants, flyers with study location and times were distributed within the exhibit hall and during interventional pulmonology-specific precourses. Potential participants were also approached for immediate participation by study personnel at the exhibit hall. Additional physicians were recruited during clinical rotations or while attending educational courses (not related to EBUS) at the M. D. Anderson Cancer Center (Houston, TX).

Survey

Bronchoscopists were initially asked to complete a multiple-choice survey to assess (1) personal practice patterns, (2) current use and acceptance of EBUS within their institution, and (3) self-perceived competence in EBUS bronchoscopy. Regarding self-perceived confidence, participants were asked to self-identify as novice, below average, average, above average, or expert. Those who self-reported a skill level of average, above average, or expert were categorized as “confident”; participants who self-reported their EBUS skill level as below average or novice were categorized as “diffident.” The estimated survey completion time was approximately 5 minutes. Content of the survey is available in the online supplement.

Simulation Scenario

After participants completed the survey, they were asked to perform a proctored high-fidelity simulated staging EBUS bronchoscopy to assess their clinical approach in endoscopic mediastinal staging. Participants used a Simbionix GI-BRONCH Mentor or Simbionix BRONCH Express simulator (3D Systems, Cleveland, OH). Although technical clarification related to the simulator was provided, participants were not given any specific coaching during the simulated examination. Before participants performed the simulated bronchoscopy, they were asked to review a brief written case vignette along with representative positron emission tomographic/computed tomographic images that showed a fluorodeoxyglucose (FDG)-avid left lower lobe mass suspicious for malignancy, with FDG avidity also present in station 11L and station 7 lymph nodes consistent with stage clinical-radiological IIIA (T1b, N2, M0) disease. The case vignette and images used are available in the online supplement.

Simulation Performance

The simulated examination was used only as a tool to evaluate the appropriateness of staging approach but not technical skills. Participants were not informed of the specific aims of the study. Performance was scored by one of two observers, and participants received a score of 1 to 4, using the following criteria:

• 1.Unable to identify or name lymph nodes or identifies lymph nodes for sampling in a nonstandard pattern.
• 2.Begins EBUS survey with radiographically abnormal N2 lymph node stations.
• 3.Performs standardized staging examination starting at contralateral mediastinal N3 lymph node stations.
• 4.Performs standardized staging examination starting at contralateral hilar N3 lymph node stations.

A score of 1 or 2 was considered “inappropriate staging” whereas a score of 3 or 4 was considered “appropriate staging.” Surveys were linked to simulation performance only by participant number, and no personal identifying information was collected.

Statistical Considerations

Data from survey respondents were captured in a Microsoft Excel spreadsheet for analysis. Descriptive statistics such as frequencies and percentages were used to summarize the characteristics of study participants. Comparisons by level of training and appropriateness of staging were made with use of a χ² test or Fisher exact test, if more appropriate.

Within each level of training, we assessed the concordance between self-confidence and appropriate staging. Results were considered concordant when confident participants performed appropriate staging or when diffident participants performed inappropriate staging. A P value less than 0.05 indicated statistical significance. Statistical analyses were performed with the use of STATA/IC version 14.0 (StataCorp LP, San Diego, CA).

Results

Sixty physicians participated in the study (22 interventional pulmonologists, 18 general pulmonologists, and 20 pulmonary fellows). Fifty-three of the participants were recruited at the ACCP 2015 Annual Meeting and an additional seven physicians were recruited during clinical rotations or while attending educational courses at the M. D. Anderson Cancer Center.

Sample Size

Initially, a sample size of 39 participants per training group was calculated based on estimations of a 30% difference in performance between interventional pulmonologists and general pulmonologists, using an α level of 0.05 and power of 0.80. Pulmonary fellows were not considered in the sample size but rather were included as part of an exploratory analysis with a maximum enrollment of 39 subjects in this group. An interim sample size review was conducted after 60 participants had completed the study. This sample size recalculation was not prespecified but was deemed prudent based on the larger-than-expected observed difference in performance. At the time of recalculation, an intergroup difference in the rate of appropriate staging of 51% was observed. On the basis of the observed difference, an α level of 0.05 and power of 0.80 could be achieved with 12 participants per group. Using the accrued sample size resulted in a post-hoc power of 0.95.

Demographics and Practice Patterns of Participants

Participant summary statistics are listed in Table 1. Yearly EBUS volume was significantly higher among interventional pulmonologists than among general pulmonologists or pulmonary fellows. There were no significant differences between groups in the use of general anesthesia or rapid onsite evaluation for EBUS bronchoscopy. Overall, interventional pulmonologists were significantly more likely to self-report as confident than were general pulmonologists or pulmonary fellows.

Table 1.

Participant characteristics by level of training

	Training			P Value*	Total (N = 60)
	Interventional Pulmonologist (n = 22)	General Pulmonologist (n = 18)	Pulmonary Fellow (n = 20)
EBUS per year, n (%)
10–19	0 (0.0)	6 (33.3)	4 (20.0)	<0.001	10 (16.7)
20–50	0 (0.0)	9 (50.0)	14 (70.0)		23 (38.3)
51 or more	22 (100)	3 (16.7)	2 (10.0)		27 (45.0)
Percentage of EBUS performed with general anesthesia, n (%)
0%	3 (13.6)	4 (22.2)	6 (30.0)	0.373	13 (21.7)
1–25%	7 (31.8)	3 (16.7)	2 (10.0)		12 (20.0)
26–50%	0 (0.0)	2 (11.1)	3 (15.0)		5 (8.3)
51–75%	2 (9.1)	1 (5.6)	3 (15.0)		6 (10.0)
76–100%	10 (45.5)	8 (44.4)	6 (30.0)		24 (40.0)
Percentage of EBUS performed with ROSE, n (%)
0%	2 (9.1)	3 (16.7)	0 (0.0)	0.631	5 (8.3)
1–25%	3 (13.6)	2 (11.1)	3 (15.0)		8 (13.3)
26–50%	1 (4.6)	2 (11.1)	1 (5.0)		4 (6.7)
51–75%	0 (0.0)	0 (0.0)	0 (0.0)		0 (0.0)
76–100%	16 (72.7)	11 (61.1)	16 (80.0)		43 (71.7)
Self-confidence, n (%)
Diffident†	0 (0.0)	3 (16.7)	8 (40.0)	0.002	11 (18.3)
Confident‡	22 (100)	15 (83.3)	12 (60.0)		49 (81.7)

Definition of abbreviations: EBUS = endobronchial ultrasound; ROSE = rapid onsite evaluation.

^*Fisher exact test P value.

^†Diffident: responding Novice/Below Average to Self-Confidence questionnaire item.

^‡Confident: responding Average/Above Average/Expert to Self-Confidence questionnaire item.

Participant Characteristics by Appropriateness of Staging

Participant characteristics that correlated with appropriate staging are summarized in Table 2. The overall rates of appropriate staging by study group were 95.5% (21 of 22) for interventional pulmonologists, 44.4% (8 of 18) for general pulmonologists, and 30.0% (6 of 20) for pulmonary fellows (P < 0.001). Seven of the 22 participants in the interventional pulmonary group were current interventional pulmonary fellows, and all of them staged correctly. Even among participants who performed appropriate staging (examination score of 3 or 4) there was variability in practice patterns, with 25.7% beginning staging examination at the mediastinal N3 lymph node stations compared with 74.2% who initiated the staging examination at the hilar N3 nodes. In all study groups, bronchoscopists who performed more than 50 EBUS bronchoscopies per year were more likely to perform appropriate staging (P < 0.001). Overall, those who self-reported as confident were more likely to perform appropriate staging (P < 0.032). However, this was heavily influenced by interventional pulmonologists who uniformly self-reported as confident, as shown in Figure 1.

Table 2.

Participant characteristics by appropriateness of staging

	Appropriate Staging		P Value*	Total (N = 60)
	No (n = 25)	Yes (n = 35)
Level of training, n (%)
Interventional pulmonologist	1 (4.5)	21 (95.5)	<0.001	22 (36.7)
General pulmonologist	10 (55.6)	8 (44.4)		18 (30.0)
Pulmonary fellow	14 (70.0)	6 (30.0)		20 (33.3)
EBUS per year, n (%)
10–19	7 (70.0)	3 (30.0)	<0.001	10 (16.7)
20–50	16 (69.6)	7 (30.4)		23 (38.3)
51 or more	2 (7.4)	25 (92.6)		27 (45.0)
Self-confidence, n (%)
Diffident	8 (72.7)	3 (27.3)	0.039	11 (18.3)
Confident	17 (34.7)	32 (65.3)		49 (81.7)

Definition of abbreviation: EBUS = endobronchial ultrasound.

^*Fisher exact test P value.

Figure 1.

Rates of appropriate staging and self-reported confidence in endobronchial ultrasound staging (defined as average, above average, or expert) within the three studied subgroups.

Of the 60 participants, 51, 9, and 0 self-identified as practicing in academic centers, community hospitals, and standalone practices, respectively. Of those who self-identified as practicing at community hospitals only two of nine (22%) staged appropriately compared with 33 of 51 (65%) of those who self-identified as practicing in academic centers. These findings should be interpreted with caution, however, as there is no agreed-on definition of academic institution or community hospital, and the self-perceived classification of participants might not reflect how one’s institution might be perceived by others.

Appropriateness of Staging by Level of Training and Confidence

Within each training group we assessed the concordance between self-reported confidence and appropriate staging, as summarized in Table 3. All interventional pulmonologists self-reported as confident, and the concordance between confidence and appropriate staging was 95.4%. Among general pulmonologists, 83.3% self-reported as confident; however, of these participants, the likelihood of appropriate staging was only 53.3%. The negative concordance within general pulmonologists was much stronger, with 100% of participants who self-reported as diffident staging inappropriately. The overall concordance between confidence and appropriate staging within this group was 61.1%. Among pulmonary fellows, 60.0% self-reported as confident, but only 25.0% of those participants staged appropriately. Of interest, pulmonary fellows who were diffident actually were more likely to stage appropriately (37.5%) than were their more confident peers. The overall concordance between confidence and appropriate staging within this group was 40.0%.

Table 3.

Appropriateness of staging by level of training and self-confidence

Level of Training by Self-Confidence	Appropriate Staging		Agreement (95% CI)*P Value	Total [N (%)]
	No [n (%)]	Yes [n (%)]
Interventional pulmonologist
Diffident†	0 (0.0)	0 (0.0)	95.4 (77.2, 99.9)	0 (0.0)
Confident‡	1 (4.6)	21 (95.4)	P = NA§	22 (100)
General pulmonologist
Diffident†	3 (100)	0 (0.0)	61.1 (35.7, 82.7)	3 (16.7)
Confident‡	7 (46.7)	8 (53.3)	P = 0.045	15 (83.3)
Pulmonary fellow
Diffident†	5 (62.5)	3 (37.5)	40.0 (19.1, 63.9)	8 (40.0)
Confident‡	9 (75.0)	3 (25.0)	P = 0.725	12 (60.0)

Definition of abbreviations: CI = confidence interval; NA = not available.

^*Exact 95% confidence intervals are provided for the proportion of agreement.

^†Diffident: responding Novice/Below Average to Self-Confidence questionnaire item.

^‡Confident: responding Average/Above Average/Expert to Self-Confidence questionnaire item.

^§κ statistic P value NA when there are insufficient data to compute the standard error.

Discussion

In this study, we found that general pulmonologists and pulmonary fellows were significantly less likely than interventional pulmonologists to perform appropriate EBUS-guided mediastinal staging. In addition, we found a lack of concordance between self-reported confidence and appropriate staging among noninterventionists performing EBUS. This is the first study that we are aware of that used high-fidelity simulation to quantify the quality in which EBUS-guided mediastinal staging occurs among diverse groups of bronchoscopists or to evaluate concordance between confidence and competence in EBUS-guided mediastinal staging.

Despite current guidelines recommending invasive mediastinal staging in patients with suspected lung cancer and radiographic findings suggesting mediastinal nodal involvement, only a minority of patients with lung cancer undergo invasive mediastinal sampling in a manner consistent with these guidelines (13, 14). Even when invasive sampling is performed, it is often not performed with enough thoroughness to meet accepted standards. An American study of surgical practices found that only 46.6% of mediastinoscopies performed for mediastinal staging in lung cancer included a biopsy, despite recommendations that at least three lymph node stations be sampled in a proper staging examination (5, 15). Our study confirmed that the significant heterogeneity in practice performance seen in mediastinoscopy also applies to EBUS-guided mediastinal staging.

When a new technology is introduced, its rapid adoption invariably results in a portion of early adopters who do not truly understand the technology. For example, early implementation of laparoscopic surgery, without proper training, initially resulted in an unexpectedly high rate of complications (16). It seems obvious that training and experience drive success in medical care. In surgical treatment of lung cancer, previous studies have shown that outcomes are better when surgery is performed by thoracic surgeons than when performed by general surgeons (17). Furthermore, surgical volume and subspecialty training have been positively correlated with patient outcome in nearly all cancer surgeries (18–20).

For endoscopic procedures, such as EBUS and endoscopic ultrasound, fewer data are available about the effects of training and volume on quality. Multiple published reports have attempted to quantify competency and training requirements for EBUS; the vast majority, however, have focused on a limited definition of competency, primarily assessing surgeons’ technical skills in performing directed diagnostic lymph node sampling (21–27). Although procedural thresholds have been developed for competency, these are relatively arbitrary, and learning curves can vary greatly between individuals (28). The ACCP released a consensus statement suggesting that professional societies move from a volume-based certification system to skill- and knowledge-based competency assessments for trainees (29). Our findings could not support or refute this approach. We found that procedural volume was strongly associated with performing a guideline-consistent EBUS evaluation, but at the same time, procedural volume and level of training were highly correlated, and we could not determine whether volume or training was responsible.

Limitations

There are some important limitations to our study that should be noted. The most obvious limitation is that we did not actually assess practice patterns but rather performance in a simulated scenario, and we cannot say with certainty that this represents the daily practices of the studied individuals. In addition, although we believe that taking advantage of a national meeting to recruit study participants provided a unique opportunity to sample a broad spectrum of physicians, we do not know how well our sampling truly mirrors the population of physicians performing EBUS in practice. Individuals attending the annual meeting are possibly more motivated to stay current on new innovations, and the deficiencies seen in our study groups might actually underestimate the true magnitude of the problem. There are many variables that are likely contributory to our findings beyond just level of training and procedural volume. For general pulmonologists, the method in which they received EBUS training (commercial course, previous fellowship, or miniapprenticeship) may affect performance. For pulmonary fellows, we suspect that individuals, within a single institution, likely perform similarly and that performance is influenced by factors such as dedicated interventional pulmonology rotations and the presence of interventional pulmonary trained staff.

Our study found a striking lack of correlation between self-rated confidence in EBUS bronchoscopy and actual likelihood of performing appropriate EBUS staging among general pulmonologists and pulmonary fellows performing this procedure. Although the vast majority of pulmonary fellowships now offer exposure to EBUS during fellowship training, our study indicated that only 30% of pulmonary fellows performed guideline-consistent staging. One could argue that as trainees advance further into fellowship training, these deficiencies are recognized and remedied before graduation and credentialing. However, if mentors within a program do not possess the core knowledge base regarding the procedures they perform, then the trainees will undoubtedly enter independent practice with a false notion of competence in their own practice. Interventional pulmonary training is potentially a tool for improving knowledge, but requiring advanced endoscopy training for EBUS credentialing is not practical. Instead, we can infer from our findings that institutions offering general pulmonary fellowships could benefit from the addition of faculty members trained in interventional pulmonary techniques, even if the general pulmonary staff currently performs EBUS within the institution. Finally, our study findings reinforce the need to develop and disseminate a standardized EBUS training curriculum that focuses on more than just the technical ability to identify and sample radiographically abnormal lymph nodes.

Conclusions

We found that general pulmonologists and pulmonary fellows were less likely than interventional pulmonologists to perform appropriate EBUS mediastinal staging. In addition, we found a lack of concordance between self-confidence and appropriate staging among general pulmonologists and pulmonary fellows. We believe that the observed disparities in staging practices among noninterventionists need to be considered as a major patient care concern. Although this study was not designed to evaluate true practice patterns and procedural quality, we believe the results are concerning enough to spur the development of a performance metric system to compare individual performance with national averages, and potentially provide bronchoscopists an opportunity to reflect on procedural deficiencies to develop targeted action plans.