Evaluation and Comparison of Performance Parameters and Impact of Telepathology and Operator Experience on Endobronchial and Endoscopic Ultrasound-Guided Fine-Needle Aspiration

Abstract

Objectives

Endobronchial ultrasound- and endoscopic ultrasound-guided fine-needle aspiration (EBUS-/EUS-FNA) are minimally invasive techniques of diagnosing and staging malignancies. The procedures are difficult to master, requiring specific feedback for optimizing yield.

Methods

Over 2 years, EBUS-/EUS-FNA cases were gathered using the institutional pathology database. Patient and specimen characteristics were collected from the pathology database and electronic medical record.

Results

In 2 years, 789 unique FNA specimens were collected (356 EBUS and 433 EUS specimens). The cohort and each subgroup had excellent performance, which was enhanced by telepathology. The discrepancy rate was satisfactorily low. Hematolymphoid neoplasms are overrepresented in discrepant EBUS cases. The malignancy rates of cytology diagnostic categories were comparable to the literature.

Conclusions

Using diagnostic yield and concordance results allow for comprehensive evaluation of the entire process of EBUS-/EUS-FNAs. This study’s findings can influence patient management, training methods, and interpretation of results, while also acting as a model for others to investigate their own sources of inadequacy, discrepancy, and training gaps.

Key Points.

• Diagnostic utility of endobronchial ultrasound- and endoscopic ultrasound-guided fine-needle aspiration is affected by operator experience, rapid on-site evaluation assessment, and the pathologist’s skills.

• This study provides a comprehensive evaluation of the entire procedure and comparison of both procedures.

• Results will be used to influence practice at our institution; other institutions may use this model to refine and monitor their practice.

Minimally invasive tissue sampling techniques have become a standard of care in current diagnostic medicine, including fine-needle aspiration (FNA) guided by endoscopic ultrasound (EUS) or endobronchial ultrasound (EBUS). Described in 2004 by Yasufuku et al,¹ EBUS-FNA is primarily a method for diagnosing and staging non–small cell lung cancers (NSCLCs), among other malignancies and medical conditions, and has a better risk profile than surgical options (thoracotomy or mediastinoscopy).² Staging is required before treatment initiation because guidelines recommend surgical resection of NSCLC without mediastinal involvement, while those with mediastinal metastases initially undergo chemotherapy.³ Diagnostic yields of EBUS-FNAs for sampling mediastinal and hilar lymph nodes are high—89% to 95% at some institutions.⁴

While EUS can complement EBUS in mediastinal staging, the EUS-FNA is primarily used to sample abdominal organs using the luminal upper (and occasionally lower) gastrointestinal tract as a conduit.⁵ The anatomy of the pancreas, the late presentation of pancreatic ductal adenocarcinoma (PDAC), and the uncertain histories of recurrent pancreatitis make this diagnosis especially difficult, particularly in those with significant morbidity. Pancreatic neoplasms are therefore commonly sampled by EUS-FNA, a minimally invasive technique described in the early 1990s,⁶ which is safer, quicker, and more cost-effective than surgical diagnosis. Almost two-thirds of the patients diagnosed with PDAC by EUS-FNA are nonresectable and are treated with neoadjuvant therapy. EUS-FNA may also investigate and sample other organs and tissues for primary tumors or metastases, including liver, adrenal gland, mesentery, retroperitoneum, and so on.⁷ As this method may be the patient’s only tissue/fluid diagnosis before beginning treatment, it is important to accurately diagnose malignancies.

As highly technical procedures, it is expected that performance parameters would vary based on operator experience. Many studies have shown the impact of interventional gastroenterologist or pulmonologist experience on adequacy rates of these procedures, but they are rarely evaluated by FNA diagnostic yield and concordance with the final “true” diagnosis. The learning curve for EBUS operation in particular can be steep and variable between operators.⁸ To our knowledge, no studies have compared the performance of EBUS and EUS proceduralists (both novice and expert) from procedures for all indications in which rapid on-site evaluation (ROSE) was performed. The data set collected was also informative on risk of malignancy, disease outcomes, discrepant cases and their causes, number of needle passes, and the role of telepathology at our institution.

Materials and Methods

This project was reviewed and approved by the institutional review board. Data were collected for procedures that were performed at our tertiary academic center between July 1, 2015, and June 30, 2017, using natural language searches in the Sunquest CoPathPlus database. This cohort included 750 EBUS specimens and 734 EUS specimens in which ROSE was performed and FNA material was collected. Each specimen may have consisted of some or all of the following: air-dried Romanowsky-stained slide, alcohol-fixed Papanicolaou-stained slides, histologic sections cut from a cell block and stained with H&E, and cell block sections with special or immunohistochemical stains applied. If multiple sites were targeted or multiple specimens were collected during a procedure, only the first specimen was included for analysis to limit bias. However, the whole case was taken into account for final designation as benign or malignant on cytologic diagnosis. For patients with multiple procedures during the study window, all subsequent specimens were removed from analysis. Patient electronic medical records and pathology reports were reviewed. Diagnostic categories were assigned for each cytology case: inadequate, nonneoplastic, benign neoplasm, indeterminate (atypical or suspicious for malignancy), and malignant.

To confirm the accuracy of cytologic diagnoses, a “final clinical diagnosis” was determined in one of three ways. The preferred method of confirmation used subsequent histopathologic specimens (biopsy specimens or resections) in all cases available (150 of 789, 19.1%). If these were unavailable, convincing ancillary diagnostic laboratory studies (including flow cytometry, bronchoalveolar cytology, bone marrow biopsy, etc) were used (6 of 789, 0.1%). Finally, if there were no convincing diagnostic laboratory studies or histopathologic specimens, clinical data were used. This included radiology, clinical impression, and serum tumor markers, among others, and required follow-up of at least 1 year after the procedure. Most cases (633 of 789, 80.2%) fell into this group. The former two methods were referred to as “definitive pathologic diagnosis.” If convincing ancillary studies or clinical follow-up were unavailable, the patient was removed from the study. The final clinical diagnoses were categorized in a similar manner as the cytologic diagnoses.

Telepathology was used by four cytopathologists (one junior with less than 3 years of experience and three seniors with more than 3 years of experience) to provide adequacy assessments in most cases. The system was developed in-house using multiple telepathology carts that were assembled to ensure mobility and standardization. Each cart was equipped with an Olympus BX43 microscope, Panasonic GP-US932HA/SYS 1080p Color Video Camera, Tandberg C20 video codec box, Extron EDID 101D emulator, NEC Multisync EA243WM LCD monitor, Polycom CX100 speakerphone, and a headset for the microscope operator. Each cytopathologist’s office/receiving site was equipped with a Cisco DX70 Video Sharing System with fully touch-based on-screen controls. In a small subset of cases, the cytopathologist provided adequacy assessment directly at the procedure site (endoscopy suites or operating rooms). If the cytopathologist was unavailable, the highest trained team member would provide adequacy assessment onsite. This may have included cytopathology fellows, pathology residents (postgraduate years [PGYs] 2-4), or cytotechnologists.

Operator experience level was determined by the number of years of institution-specific experience—those with less than 3 years of experience were deemed “junior operators” and those with 3 or more years were “senior operators.” EBUS operators, typically pulmonologists, were predominantly junior operators; there were two senior and six junior operators, although more than 90% of procedures were performed by the two senior and three of the junior operators. EUS had a more even distribution; there were three senior and three junior operators, but more than 95% of procedures were performed by two senior and two junior individuals. Documentation of senior operators assisting their junior counterparts was available for EUS, as recorded in the procedure notes, and was extrapolated, but it was not available or reliably documented for EBUS procedures. The summary statistics generated were grouped by procedure (EBUS/EUS) and by operator experience. Comparisons of continuous variables were conducted with t tests (two-tailed), and categorical variables were compared with χ ² tests without correction. Performance parameters of positive predictive value (PPV), negative predictive value (NPV), sensitivity, specificity, and accuracy (proportion of matching cytologic and final diagnoses) were calculated for the entire sample, grouped by procedure, by operator experience, and by both. For the purposes of the performance parameters, malignant (for cytologic and final results) and indeterminate (for cytologic results, including atypical and suspicious) diagnostic categories were considered positive outcomes, while nonneoplastic and benign neoplasm (for cytologic and final results) diagnostic categories were considered negative outcomes.

Analyses were conducted with R (version 3.6.1; R Foundation). P values were calculated using the χ ² test without Yates’s correction and are two-tailed unless otherwise noted.

Results

Specimen and Procedure Characteristics

Overall, 789 unique qualifying specimens (356 EBUS-FNA specimens and 433 EUS-FNA specimens) were collected during the study timeframe with adequate follow-up. The most frequently targeted tissue was lymph node tissue Table 1. The most common cytologic diagnostic category was malignant (343, 43.5%), followed by nonneoplastic (303, 38.4%), inadequate (70, 8.9%), indeterminate (61, 7.7%), and benign neoplasm (12, 1.5%). The indeterminate cases were further divided into 50 cases designated “atypical” (6.3% of all cases) and 11 cases designated “suspicious” (1.4%). The final clinical diagnostic categories were malignant (407, 51.6%), nonneoplastic (360, 45.6%), and benign neoplasm (22, 2.8%; Table 2). It should be noted that no cases were “inadequate” or “indeterminate” at final diagnosis; these cases were previously removed from analysis for lacking histopathologic specimens or convincing ancillary studies. Junior operators collected fewer specimens than senior operators (290 vs 499 specimens, 36.8% vs 63.2%). The repeat procedure rate was satisfactorily low (33 cases, 4.2%). A significant minority (156, 19.8%) of the cases went forward to “definitive pathologic diagnosis”—resection (102 of 789, 12.9%), biopsy (40, 5.1%), autopsy (6, 0.8%), repeat procedure (2, 0.3%), or other cytologic specimens (6, 0.8%).

Table 1.

Targeted Organ/Tissue^a by Procedure

Characteristic	No. (%)
EBUS (n = 356)
Lymph node	291 (81.7)
Lung	52 (14.6)
Para/pretracheal	6 (1.7)
Mediastinum	5 (1.4)
Endobronchial	2 (0.6)
EUS (n = 433)
Pancreas	139 (32.1)
Lymph node	132 (30.5)
Liver	40 (9.2)
Stomach	29 (6.7)
Adrenal gland	17 (3.9)
Duodenum	13 (3.0)
Retroperitoneum	12 (2.8)
Esophagus	11 (2.5)
Lung	10 (2.3)
Bile duct	7 (1.6)
Mediastinum	6 (1.4)
Rectum	6 (1.4)
Porta hepatis, NOS	4 (0.9)
Colon	3 (0.7)
Jejunum	3 (0.7)
Spleen	1 (0.2)

NOS, not otherwise specified.

^aIncludes soft tissue surrounding the organ (eg, perirectal soft tissue).

Table 2.

Comparison of Cytologic Diagnosis and Final Clinical Diagnosis Categories

	EBUS (n = 356), No. (%)		EUS (n = 433), No. (%)
Diagnostic Category	Cytologic Diagnosis	Final Clinical Diagnosisa	Cytologic Diagnosis	Final Clinical Diagnosisa
Inadequate	51 (14.3)	0 (0.0)	19 (11.3)	0 (0.0)
Nonneoplastic	167 (46.9)	201 (56.5)	136 (31.4)	159 (36.7)
Benign neoplasm	0 (0.0)	2 (0.6)	12 (2.8)	20 (4.6)
Indeterminate (atypical/ suspicious)	12 (3.4)	0 (0.0)	49 (11.3)	0 (0.0)
Malignant	126 (35.4)	153 (43.0)	217 (50.1)	254 (58.7)

EBUS, endobronchial ultrasound; EUS, endoscopic ultrasound.

^aBased on ancillary testing, biopsy, or resection.

For EBUS-specific procedure and specimen characteristics, please refer to Tables 1 and 2. Junior operators performed 170 (47.8%) EBUS-FNA procedures, while senior operators performed 186 (52.3%). Very few EBUS cases performed by a junior operator were known to have had assistance from senior operators, but these data were not reliably documented in the procedure note and therefore not extrapolated for this study. Repeat procedure was required in 11 (3.1%) cases. “Definitive pathologic diagnosis” was available in 59 (16.6%) cases, with the following distribution—resection (30 of 356, 8.4%), biopsy (20, 5.6%), autopsy (3, 0.8%), repeat procedure (1, 0.3%), or other cytologic specimens (5, 1.4%).

For EUS-specific procedure and specimen characteristics, refer to Tables 1 and 2. Compared with EBUS, a higher proportion of EUS cases was malignant (P < .0001). Junior operators collected 120 (27.7%) of the 433 EUS-FNA specimens, significantly less than the junior EBUS operators (47.8%, P < .0001). Senior operators assisted 28 cases that had a junior operator assigned as the primary provider (of 120 junior-operated EUS cases, 23.3%). When these cases were excluded, only minor differences in the performance parameters were seen that were statistically insignificant, as described in the performance section below.

Twenty-two (5.1%) procedures required repeat sampling, not a significantly different rate from the EBUS rate (P = .1645). Ninety-seven (22.4%) specimens had subsequent “definitive pathologic diagnosis,” a higher fraction than EBUS (P = .0408). There were 72 (16.6% of 433) resection cases, also a higher rate than EBUS (P = .0006). The remaining final clinical diagnoses were determined by biopsy (20, 4.6%), autopsy (3, 0.7%), repeat procedure (1, 0.2%), and other cytologic specimens (1, 0.2%).

Performance

The cytologic diagnoses were compared with the final clinical diagnoses (Table 2) to calculate sensitivity, specificity, PPV, and NPV. The entire cohort had excellent performance parameters, with an overall accuracy of 0.9234 Table 3. For all procedures, both seniors and juniors had excellent performance parameters as well, although junior EUS providers had slightly higher specificity, PPV, and accuracy. When the junior-operated EUS cases that had senior operator assistance (28 cases) were removed from this evaluation, the performance parameters were not significantly affected. Sensitivity rose from 0.96 to 0.98, specificity and PPV remained at 1.00, NPV rose from 0.91 to 0.96, and accuracy rose from 0.96 to 0.99.

Table 3.

Performance Parameters Without Inadequate Specimens and Indeterminate Specimens Considered Malignant

Characteristic	Sensitivity	Specificity	PPV	NPV	Accuracy
All	0.9491	0.9077	0.9256	0.9365	0.9234
All juniors	0.9467	0.9626	0.9726	0.9279	0.9494
All seniors	0.9506	0.8807	0.8988	0.9412	0.9089
EBUS	0.9366	0.9693	0.9638	0.9461	0.9475
EBUS juniors	0.9265	0.9467	0.9403	0.9342	0.9371
EBUS seniors	0.9459	0.9886	0.9856	0.9560	0.9568
EUS	0.9562	0.8457	0.9057	0.9527	0.9056
EUS juniors	0.9634	1.0000	1.0000	0.9143	0.9649
EUS seniors	0.9527	0.8077	0.8656	0.9292	0.8829

EBUS, endobronchial ultrasound; EUS, endoscopic ultrasound; NPV, negative predictive value; PPV, positive predictive value.

Passes

During the procedure, the ROSE cytopathology team designated each pass as (1) “adequate” to reach final clinical diagnosis and perform ancillary studies, (2) “inadequate,” or (3) placed directly in media. These media included RPMI, formalin, and saline for cell block material or future ancillary studies such as flow cytometry, cytogenetics, and microbiology cultures. Overall, an average of 4.7 (confidence interval [CI], 4.6-4.9) total passes were taken in the entire cohort. Of these, 2.1 (CI, 2.0-2.2) were deemed adequate by the ROSE team, 1.6 (CI, 1.5-1.8) were inadequate, and 1.0 (CI, 0.9-1.1) were placed directly in the media. There was no significant difference in adequate passes between EBUS and EUS cases (2.0 vs 2.2, P = .2151), but EBUS cases had higher average rates of total passes (5.1 vs 4.5, P < .0001), inadequate passes (1.9 vs 1.4, P = .0006), and passes placed directly in media (1.1 vs 0.9, P = .0027) Table 4. When separated by operator experience, the only significant differences were between EUS junior and senior operators for average total passes (juniors 4.0 vs seniors 4.6, P = .0004) and average inadequate passes (1.1 vs 1.6, P = .0131; Table 5).

Table 4.

Mean Passes per Specimen in Each Diagnostic Category as Designated by the ROSE Cytopathology Team (With 95% Confidence Intervals)

Characteristic	EBUS, Mean (95% CI)	EUS, Mean (95% CI)	P Valuea
EBUS, EUS, and all procedures
Total	5.1 (4.9-5.3)	4.5 (4.3-4.6)	<.0001
Adequate	2.0 (1.9-2.2)	2.2 (2.0-2.3)	.2151
Inadequate	1.9 (1.7-2.1)	1.4 (1.3-1.6)	.0006
Permanentb	1.1 (1.0-1.3)	0.9 (0.8-1.0)	.0027

CI, confidence interval; EBUS, endobronchial ultrasound; EUS, endoscopic ultrasound; ROSE, rapid on-site evaluation.

aBold values indicate statistical significance.

^bIndicates passes placed directly in media for permanent sections, flow cytometry, molecular studies, and so on.

Table 5.

Mean Passes per Specimen in Each Diagnostic Category by Operator Experience as Designated by ROSE Cytopathology Team

Characteristic	Junior, Mean (95% CI)	Senior, Mean (95% CI)	P Valuea
EBUS procedure
Total	5.1 (4.8-5.4)	5.0 (4.7-5.3)	.6247
Adequate	2.0 (1.8-2.2)	2.1 (1.9-2.3)	.3807
Inadequate	2.0 (1.7-2.3)	1.8 (1.6-2.1)	.5680
Permanentb	1.2 (1.0-1.4)	1.1 (0.9-1.3)	.4723
EUS procedure
Total	4.0 (3.7-4.3)	4.6 (4.4-4.8)	.0004
Adequate	2.0 (1.8-2.2)	2.2 (2.1-2.4)	.1443
Inadequate	1.1 (0.8-1.4)	1.6 (1.4-1.8)	.0131
Permanentb	0.9 (0.7-1.0)	0.9 (0.8-1.0)	.9441
All procedures
Total	4.7 (4.4-4.9)	4.8 (4.6-4.9)	.4188
Adequate	2.0 (1.8-2.1)	2.2 (2.0-2.3)	.0675
Inadequate	1.6 (1.4-1.8)	1.7 (1.5-1.8)	.7089
Permanentb	1.1 (0.9-1.2)	0.9 (0.8-1.1)	.1365

CI, confidence interval; EBUS, endobronchial ultrasound; EUS, endoscopic ultrasound; ROSE, rapid on-site evaluation.

aBold values indicate statistical significance.

^bIndicates passes placed directly in media for permanent sections, flow cytometry, molecular studies, and so on.

Adequacy

Inadequacy was investigated using the cytologic diagnosis category of “inadequate.” EBUS cases had a higher rate of inadequacy than EUS cases, 51 (14.3%) vs 19 (4.4%; P < .0001). Overall, the adequacy rate was 91.1% (719 of 789). When comparing junior vs senior operators, there was no statistical difference of adequacy rates within the EBUS procedure, the EUS procedure, and all procedures. The junior EBUS adequacy rate was 84.1% (143 of the 170 junior-obtained EBUS specimens), which is not significantly different from the senior EBUS adequacy rate of 87.1% (162 of 186; P = .42). For the EUS procedures, junior operators had an adequacy rate of 95.8% (115 of 120) while seniors had an adequacy rate of 95.9% (300 of 313; P = .7003). When combining both procedures, the junior adequacy rate was 88.6% (257 of 290), and the senior adequacy rate was 92.6% (462 of 499; P = .0590).

Discrepancy

Discrepancy was assessed by comparing the cytologic diagnostic class with the final clinical diagnostic class for the cases deemed adequate on cytology (nonneoplastic, benign neoplasm, indeterminate, and malignant). Inadequate specimens that had a confirmed final clinical diagnosis later were excluded and removed from subsequent discrepant evaluations. Indeterminate (atypical or suspicious) specimens that were designated malignant or benign neoplasm on final clinical diagnosis were considered concordant, while nonneoplastic final clinical diagnoses were discrepant and will be referred to as “discrepant indeterminate” cases.

Of the 14 (3.9%) discrepant EBUS cases, four (2.8%) were indeterminate (atypical or suspicious for neoplasm) cases and 10 were truly discrepant cases. “Truly discrepant” cases are those that were initially placed into one diagnostic category but later diagnosed as a different category (eg, initially “nonneoplastic” found to be “malignant” on resection or subsequent biopsy specimen). All truly discrepant cases carried the cytologic diagnosis of “nonneoplastic.” Nine targeted a lymph node and one targeted the lung. The final clinical diagnoses included squamous cell carcinoma (n = 4), classic Hodgkin lymphoma (n = 2), small lymphocytic lymphoma Image 1, Castleman disease, diffuse large B-cell lymphoma, and neuroendocrine carcinoma. Although hematolymphoid neoplasms (HLNs) accounted for only 12 (3.4%) of the 356 total EBUS diagnoses, the proportion of HLNs incorrectly diagnosed as benign was quite high—5 of 12 or 41.7%. This is significantly higher than the discrepancy rate of non-HLN diagnoses (5 of 344, 1.5%; P < .0001) and higher than the squamous cell carcinoma discrepancy rate (4 of 54, 7.4%; P = .0041), which was the next most commonly missed diagnosis.

Image 1.

Images of a lymph node aspiration that was originally categorized as “nonneoplastic” and later shown to be involved by small lymphocytic lymphoma. A, Diff-Quik, ×400. B, Papanicolaou, ×400.

Of the 32 (7.4%) discrepant EUS cases, most (23 of 32, 71.9%) were discrepant indeterminate cases; there were only 9 (28.1%) of 32 truly discrepant cases, which accounted for 2.1% of all EUS specimens. All cases had cytologic diagnoses of nonneoplastic. Six targeted the pancreas, one the porta hepatis, one the liver, and one the adrenal gland. The diagnoses included adenocarcinoma (n = 4), intraductal papillary mucinous neoplasm (n = 2), mucinous cystic neoplasm, myeloid sarcoma, and adrenal adenoma. There were similar rates of EUS and EBUS discrepant indeterminate cases within the “indeterminate” category—23 (46.9%) of 49 total indeterminate EUS cases vs 4 (57.2%) of 11 total indeterminate EBUS cases (P = .5241). However, the rate of discrepant indeterminate specimens of total specimens was higher for EUS (23 of 433, 5.3%) than for EBUS (4 of 356, 1.1%; P = .0013). There was no significant difference between truly discrepant EBUS and EUS cases of total cases (EBUS, 10 of 356, 2.8%; EUS, 9 of 433, 2.0%; P = .5054), but the rates of truly discrepant within the discrepant category were significantly different (EBUS, 10 of 14, 71.4%; EUS, 9 of 32, 28.1%; P = .0061).

EBUS junior operators obtained three discrepant indeterminate specimens and five truly discrepant specimens. EBUS senior operators obtained one discrepant indeterminate specimen and five truly discrepant specimens. EUS junior operators had no discrepant indeterminate specimens and 3 truly discrepant specimens, while seniors had 23 and 3 specimens, respectively. There was no significant difference between junior and senior operators for truly discrepant EBUS, discrepant indeterminate EBUS, or truly discrepant EUS specimens (data not shown). Conversely, junior EUS operators had no discrepant indeterminate cases in their 120 total, while seniors had 23 in their 313 total (7.4%, P < .0001). Upon pooling experience levels in both procedures, there was no significant difference between junior and senior rates of truly discrepant cases (data not shown).

Telepathology and Adequacy Assessment

Overall, telepathology was used in 565 (71.7%) cases, and in all these cases, the evaluation and interpretation were carried out by four cytopathologists. Utilization rates were 68.8% (245 of 356) for EBUS cases and 73.9% (320 of 433) for EUS cases, which were not significantly different (P = .1151). The telepathology utilization rate for EBUS procedures was 71.2% by junior operators compared with 66.7% by senior operators. For EUS procedures, the utilization rate was 75.0% by junior operators compared with 73.5% by senior operators. These differences were not statistically significant (data not shown).

Adequacy was increased with the use of telepathology in the following subgroups: all cases, all EBUS cases, all EUS cases, EBUS senior operators, and EUS junior operators Table 6. There was no significant difference in adequacy with the use of telepathology for EBUS junior operators or EUS senior operators. Likewise, concordance was increased with the use of telepathology in the same subgroups described above (Table 6).

Table 6.

Impact of Telepathology Utilization on Concordance Between Cytologic Diagnosis and Final Clinical Diagnosis (Including Inadequate Samples as Discrepant) and Adequacy

	Concordance With Final Clinical Diagnosis			Adequacy
Characteristic	Without Telepath, %	With Telepath, %	P Valuea	Without Telepath, %	With Telepath, %	P Valuea
All	70.6	75.6	.0003	70.4	85.9	.0003
EBUS	73.9	85.3	.0097	78.4	89.0	.0082
EBUS juniors	73.5	81.8	.2227	77.6	86.8	.1361
EBUS seniors	74.2	88.7	.0112	79.0	91.1	.0203
EUS	82.3	90.3	.0231	92.0	96.9	.0308
EUS juniors	83.3	95.6	.0277	86.7	97.8	.0156
EUS seniors	81.9	88.7	.1468	94.0	96.5	.3190

EBUS, endobronchial ultrasound; EUS, endoscopic ultrasound.

aBold values indicate statistical significance.

Junior cytopathologists provided adequacy assessment in 197 (37.4%) of 527 cases, while seniors provided it in 330 (62.6%) cases. There was no statistical difference between the inadequacy rates of junior and senior cytopathologists when both procedures were grouped together (7.1% vs 7.0%, P = .9525) or separated into EBUS (8.3% vs 15.6%, P = .1531) and EUS procedures (5.9% vs 1.9%, P = .0718; Table 7).

Table 7.

Distribution of Telepathology and Nontelepathology Cases Based on Adequacy Assessment Provider

Team Member	Total Cases, No. (%)	Inadequate Cases,a No. (%)	Inadequate Cases (% of Total), No. (%)
EBUS procedure	(n = 339)	(n = 53)
Nontelepathology	121b (35.7)	26 (21.5)	121 (3.5)
Cytotechnologist	7 (2.1)	0 (0)	0 (0)
Resident	35 (10.3)	7 (20.0)	7 (2.1)
Fellow	43 (12.7)	12 (27.9)	12 (3.5)
Junior cytopathologist	16 (4.7)	6 (37.5)	6 (1.8)
Senior cytopathologist	19 (5.6)	1 (5.3)	1 (0.3)
Telepathology	218 (64.3)	27 (12.4)	27 (8.0)
Junior cytopathologist	96 (1.8)	8 (8.3)	8 (2.4)
Senior cytopathologist	122 (36.0)	19 (15.6)	19 (5.6)
EUS procedure	(n = 438)	(n = 18)
Nontelepathology	129 (29.5)	8 (0.6)	8 (1.8)
Cytotechnologist	1 (0.2)	0 (0)	0 (0)
Resident	39 (8.9)	2 (5.1)	2 (0.5)
Fellow	49 (11.2)	5 (10.2)	5 (1.1)
Junior cytopathologist	13 (10.1)	1 (7.7)	1 (0.2)
Senior cytopathologist	27 (3.0)	0 (0)	0 (0)
Telepathology	309 (70.5)	10 (3.2)	10 (2.3)
Junior cytopathologist	101	6 (5.9)	6 (1.4)
Senior cytopathologist	208	4 (1.9)	4 (0.9)

EBUS, endobronchial ultrasound; EUS, endoscopic ultrasound.

^aPercentage calculated out of total cases in that subgroup (eg, 7 of 35 or 20% of resident EBUS cases were inadequate).

^bOne (0.8%) EBUS case with no provider data available.

In 260 (28.3%) cases where telepathology was unavailable or not used, the adequacy assessments were provided by cytopathologists, cytotechnologists, residents (PGYs 2-4), or cytopathology fellows (Table 7). Since cytotechnologists provided adequacy assessments in only 8 (3%) of 260 nontelepathology cases, they were excluded from comparison analysis. Residents and fellows had higher numbers and rate of inadequate cases compared with cytopathologists (26 vs 8), but the difference did not reach statistical significance (EBUS: P = .6107, EUS: P = .2372) due to the relatively low number of cases interpreted by cytopathologists (Table 7).

Risk of Malignancy

Each diagnostic category has different intrinsic risks of malignancy (RoMs).⁹ Our data showed a 20.0% RoM for inadequate specimens, 6.6% for nonneoplastic, 0.0% for benign neoplasm, 52.4% for indeterminate, and 99.7% for malignant. To facilitate comparison with published data, the “indeterminate” category was split into their original “atypical” or “suspicious” cytologic diagnosis. The overall RoM was 46.0% (23 of 50) for atypical and 72.7% (8 of 11) for suspicious cases. See Figure 1 for procedure-specific RoM. There was no significant difference between EBUS and EUS RoMs for each diagnostic category (data not shown).

Figure 1.

Risk of malignancy by diagnostic category, separated by procedure. EBUS, endobronchial ultrasound; EUS, endoscopic ultrasound.

When separated by operator experience, only cases within the “indeterminate” category reached statistical significance Figure 2. For EUS cases, atypical cases had a RoM of 100% (9 of 9) with junior operators and a RoM of 31.3% (10 of 32) with senior operators (P = .0016). No such difference was noted for EBUS cases—50.0% (1 of 2) with junior operators and 42.9% (3 of 7) with senior operators (P = .8577). For suspicious cases, junior EUS operators had no cases designated as “suspicious,” while senior operators had a RoM of 62.5% (5 of 8; unable to calculate P value). Again, no such difference was seen in suspicious EBUS cases, as the RoMs with both junior and senior operators were 100.0% (2 of 2 and 1 of 1, respectively). EBUS junior operators had a lower RoM in the atypical category than the EUS junior operators—50.0% (1 of 2) vs 100.0% (9 of 9; P = .0261). No other comparisons between operator levels and procedures reached statistical significance (data not shown). As a whole, junior operators had a higher RoM than senior operators for atypical cases—90.9% (10 of 11) vs 33.3% (13 of 39; P = .0007). The RoM for suspicious cases did not have enough cases to reach statistical significance—100.0% (2 of 2) vs 66.7% (6 of 9; P = .3384).

Figure 2.

Risk of malignancy by diagnostic category, separated by operator experience and procedure. EBUS, endobronchial ultrasound; EUS, endoscopic ultrasound; Jr, junior; Sr, senior.

Discussion

This study provides an important insight into the diagnostic utility of EBUS- and EUS-FNA as a whole. These procedures are mainstays of current oncologic diagnosis and staging and allow for tissue sampling in patients with significant comorbidities. As technically challenging procedures with long learning curves, understanding the diagnostic performance parameters and limitations is paramount in medical decision making. How cytologic diagnoses from these specimens correlate with the final clinical diagnosis is a factor of the operator’s experience and skills, the ROSE team’s ability to assess for adequacy, and the signing-out pathologist’s skills. This study is (to our knowledge) the first to investigate EUS-/EBUS-FNA for all indications in this manner at a single institution. While this study provides insight into only one institution, we believe it provides a framework for others to reproduce.

The most frequently targeted tissues were lymph node and lung for EBUS and pancreas and lymph node for EUS, reflecting the most common indication for each procedure—diagnosis or staging for NSCLC and diagnosis or staging for pancreatic lesions, respectively. More EUS cases progress to surgical resection than EBUS cases, which may reflect pancreatic adenocarcinoma guidelines vs NSCLC guidelines (the two most frequent diagnoses for each procedure, respectively).

Both procedures have acceptably low rates of repeat procedure. The inadequacy rate was higher in EBUS cases than EUS (14.3% vs 4.4%), but there was no significant difference between levels of experience between and among each procedure group. The higher inadequacy rate from EBUS specimens may reflect the inherent technical difficulty of the EBUS procedure. EBUS and EUS had similar low rates of truly discrepant cases and cases reported “atypical” or “suspicious for malignancy” later identified as nonneoplastic (“discrepant indeterminate”). The overall cohort has impressive performance parameters, with an accuracy of 0.9234.

Senior-level operators for both procedures each performed around 50 procedures per year, and junior operators each performed around 14 (EBUS) to 20 (EUS) procedures per year if averaged equally among all providers. These numbers are not reflective of the reality at this institution; a handful of providers performed 90% to 95% of the procedures, with a smattering of other providers performing the remainder. The American Society for Gastrointestinal Endoscopy recommends 150 supervised EUS procedures with at least 75 evaluating the pancreaticobiliary system and at least 50 with FNA to reach competency.¹⁰ A previously published systematic review revealed that the minimum number of procedures to perform before “comprehensive competency” is 225, and the average is about 330 EUS procedures per trainee.¹⁰ Continued competency may require at least 10 procedures per year.⁴ Sensitivity of EBUS-FNA operators has been shown to consistently and dramatically improve up to 50 completed procedures, with a more gradual improvement up to 200 completed cases.³ Each year of training has been shown to increase yield and decrease inadequacy rates.⁴ One systematic review emphasized the variability in the number of procedures performed before a thoracic surgeon or respiratory physician reached 80% accuracy, ranging from 10 and 529 procedures and a mean of 91.2 procedures. After removing one study from this review, the mean dropped to 36.5 procedures.¹¹ EBUS-FNAs of mediastinal or hilar nodes in one study had a pooled sensitivity of 67.4% and an acceptable failure rate of 10% to 15% or lower. Two meta-analyses described better performance—pooled sensitivity of 88% to 93% and pooled specificity of 100%. Our data are consistent with the latter data.^3,4,8

The EUS-FNA’s use to diagnose solid pancreatic lesions has a reported diagnostic accuracy of 78% to 95%, sensitivity of 85% to 95%, and specificity of 95% to 98%.¹² There are unique limitations to sampling the pancreas. Variant anatomy and obfuscation of normal anatomy by chronic pancreatitis are only a few of the contributors to this difficulty, which has been shown to cause false negatives.¹² In addition, chronic pancreatitis can cause cytopathologic or histopathologic false negatives and false positives.^12,13 While these data are useful for tailoring pancreas FNA education, having data on EUS-FNAs performed on all targets gives useful feedback to the gastroenterologists and the cytopathologists. In this respect, the EUS operators at our institution have comparable sensitivity and accuracy for all EUS-FNAs to others’ published parameters. Compared with others, our specificity is marginally low (84.6%). Upon investigation, this appears to be a result of requiring atypical and suspicious cases to be classified as “malignant” for statistical purposes. For example, the senior EUS operators had 23 false positives, which were all originally designated atypical/suspicious on cytology and later deemed “nonneoplastic.” Therefore, the marginally low specificity and the higher rate of “atypical/suspicious” cytologic diagnoses on EUS-FNA specimens reveal a potential overuse of the indeterminate category in this group.

ROSE aids in confirming the location of the FNA needle and confirming adequate cellularity for diagnosis and ancillary studies. In 2000, Erickson et al¹⁴ claimed that absence of ROSE in EUS-FNA is associated with a 10% to 15% reduction in definitive diagnosis for pancreatic malignancies, as well as more needle passes, longer procedure time, and higher risk to the patient, which has been corroborated in subsequent studies, although cost is driven up by pathologists’ time and skills.^12,13,15 Telepathology (or telecytology) provides an opportunity to decrease cost while maintaining ROSE presence. It has been shown to be similar to live microscopy.^16-18 Telepathology was used in most procedures in this study. Our study is one of the few studies indicating telepathology significantly improves specimen adequacy and concordance with final clinical diagnosis. One of the reasons that most likely contributed to this improved performance is adequacy assessment by the cytopathologists on all telepathology cases, whereas most of the nontelepathology cases were evaluated by cytotechnologists, residents, and fellows (Table 6). Furthermore, we also compared the performance of junior and senior cytopathologists and found no significant difference in the inadequacy rates between them.

Important contributors to false-negative EBUS results at this institution are HLNs, which were disproportionately represented in our truly discrepant cases. HLNs can be challenging to diagnose on cytology alone, often requiring evaluation of architecture and ancillary testing such as flow cytometry or cytogenetics. Lymphomas are being diagnosed more routinely on limited samples—at one institution, the rate of lymphomas that were diagnosed on FNA or core needle biopsy rose from 15% in 1991 to 70% in 2016.¹⁹ While diagnosis of high-grade HLN such as lymphoblastic lymphoma may be more easily accomplished, diagnosis of small mature lymphocytic neoplasms, neoplasms with rare neoplastic cells (classic Hodgkin lymphoma, T-cell/histiocyte-rich large B-cell lymphoma, etc), and rare reactive/pseudoreactive diseases such as Castleman disease are difficult to diagnose on limited samples. Requesting additional cytologic material for ancillary testing or recommending excisional biopsy can assist and avoid delay in diagnosis. However, ancillary assays such as flow cytometry are also limited by low cellularity. For example, a lymph node that was later proven to be involved by small lymphocytic lymphoma (Image 1) had poorly preserved morphology (simply demonstrating a monomorphic-appearing population of small lymphocytes) and a paucicellular cell block, limiting the immunohistochemical stains that could be applied. Unfortunately, because only a small amount of tissue could be aspirated, no tissue was able to be submitted for flow cytometric immunophenotyping, and the case was signed out as nonneoplastic but limited by low cellularity with a comment recommending resampling if a small B-cell lymphoma was suspected. When HLN is in the differential at the time of ROSE, diagnostic potential can be maximized by obtaining more passes, with several dedicated for flow cytometry (and other ancillary studies), and using careful smear technique to preserve morphology. The different diagnostic categories (nonneoplastic, indeterminate, benign neoplasm, malignant, inadequate) inherently come with varying levels of malignancy risk. On average, each cytologic diagnostic category had RoM similar to those in the literature, although most published risks and outcomes are for one disease only.^9,20-23 One study of pancreatic masses found that 65.5% of “atypical” specimens, 92.3% of “suspicious” specimens, and 99.3% of “malignant” specimens were later confirmed as malignant.⁹ For the most part, the only categories in which RoM differed between operator levels and procedures were “atypical” and “suspicious,” although the low number of cases within this category (while arguably beneficial for practice as a whole) makes interpretation of these differences difficult. The RoM for inadequate specimens can be particularly useful in future practice at this institution, as it may influence decisions on future procedures. For example, if an EBUS senior operator has an inadequate sample, they know that it has almost a 30% chance of being malignant and may then decide to obtain another EBUS specimen or an excisional biopsy specimen.

This study had several limitations. First, multiple providers obtained this study’s samples, introducing variation in operator technique and skill. However, this variation reflected the operator skills and was important for comparison of junior- and senior-level providers. Second, this study is limited to a single academic tertiary care center, so the results may not reflect other institutions’ practices. Another possible limitation is the default categorization of cases with no repeat procedures, surgeries, diagnostic imaging, or other clinical findings into the “nonneoplastic” category. Follow-up for all patients was at least 1 year to limit this effect, but we acknowledge that some malignant neoplasms have indolent courses and may have been miscategorized by our system. Finally, some of the cases primarily assigned to junior operators were partly performed by their senior counterparts. These data were available for EUS, in which 28 procedures were assisted by a senior operator. Statistical analysis performed with these cases removed did not show any significant change. However, no such information could be extrapolated from the EBUS procedure notes.

In summary, this study provides a unique insight into performance parameters and outcomes of EBUS- and EUS-FNA procedures regardless of indication or target. This study may be used to support or influence current training methods for those performing these procedures. It also provides a baseline for institutional use to optimize diagnostic value of cytologic material and finally a quality metric to improve the overall performance of these procedures.

This research was supported by the National Institutes of Health’s National Center for Advancing Translational Sciences (grant UL1TR002494). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health’s National Center for Advancing Translational Sciences.