Learning Electromagnetic Navigational Bronchoscopy and Percutaneous Transthoracic Needle Biopsy (LEAP): A Pilot Study

Abstract

Background

Prior studies in pulmonology have examined the validity of procedural training tools, however, translation of simulation skill acquisition into real world competency remains understudied. We examine an assessment process with a simulation training course for electromagnetic navigational (EMN) bronchoscopy and percutaneous transthoracic needle aspiration (PTTNA).

Methods

A cohort study was conducted by subjects using EMN bronchoscopy and PTTNA. A procedural assessment tool was developed to measure basic competency for EMN bronchoscopy and PTTNA at three different time points: first simulation case, final simulation case upon reaching a competent score, and at their first live case. The assessment tool was divided into four domains (total score 4–16, competency ≥ 12) with each domain requiring a passing score (1–4, competency ≥ 3.0). Complication, and procedural time were collected during their first live case.

Results

Twenty-two serial procedures (12 EMN bronchoscopies, 10 EMN PTTNA) were observed by fourteen subjects. The mean first simulation score for EMN bronchoscopy (4.66 ±0.89) improved after cadaver simulation (12.67 ±0.89, median 3 simulations attempts). The subjects’ mean score for their first live case was 13.0 ±0.85 (self-reported score 12.5 ±1.07).

For EMN PTTNA, the mean first simulation score (4.3 ±2.40) improved after cadaver simulation (12.6 ±1.51, median 3 simulation attempts). The subjects’ mean score for their first live PTTNA case was 12.5 ±2.87 (self-reported score 12.1 ±1.05). There was only one minor complication.

Conclusion

Learning EMN bronchoscopy/PTTNA is feasible using a structured simulation course with an assessment tool.

INTRODUCTION

The growth of technology has generated an increasing number of procedures available to physicians. While many physicians learn some procedures during their residency/fellowship training, some procedures may require mastery outside of this formal training environment. Examples of two newer procedures, developed for pulmonologists, include electromagnetic navigational (EMN) bronchoscopy and EMN percutaneous transthoracic needle aspiration (PTTNA). For practitioners learning new procedures, continuing medical education, and post graduate simulation courses may be essential to developing competency.^1–4 Prior tools have been developed to assess competency in basic bronchoscopy^4–6 and endobronchial ultrasound guided transbronchial needle aspiration.³ In addition, the CHEST expert panel and Skills-based Working Group of the American Thoracic Society Education Committee recently recommended that the optimal curriculum for pulmonary medicine training should include simulation.^6–7 However, there are currently no data on the effectiveness of simulation for EMN bronchoscopy or validated competency metrics. A more recent expansion of EMN technology has introduced EMN PTTNA. This technology shows promising results in a single center pilot study for diagnosing peripheral pulmonary nodules (PPN).⁸ In our pilot study, we aim to develop an assessment tool coupled with a simulation EMN course using structured feedback based on this assessment tool.

MATERIALS AND METHODS

This study was approved by the Johns Hopkins Medical Institution Institutional Review Board (IRB00034982). We performed a prospective cohort study examining the technical skill of pulmonary practitioners in the performance of EMN bronchoscopy and PTTNA (Veran Medical Technologies, St Louis, MO) for the biopsy of PPN on cadaver simulation models followed by live cases at the practitioner’s medical center

Assessment Tool

A procedural assessment tool was developed/by four content expert investigators (HL, LY, DF, SA) (experience >20 EMN bronchoscopies/PTTNAs) in the field of interventional pulmonology with prior experience in biopsy of PPN via EMN bronchoscopy and PTTNA (Table 1). The assessment tool was based on a modification of the Dreyfus model of skill acquisition and a validated assessment tool used in opthalmic surgery.^{9, 10} The assessment tool contains two separate sections: one for EMN bronchoscopy and a second for EMN PTTNA. EMN bronchoscopy skills were divided into four domains: 1) pre-procedure evaluation and planning, 2) equipment setup and registration, 3) navigation to target lesion and 4) performance of biopsy. Each domain consisted of several different tasks. Each task was graded on a scale of 1–4, the mean task scores for each domain was used for a total procedural score of 16. Subjects received one point if they required significant assistance to perform the majority of the task, two points if they required minimal, but critical assistance, three points if they required minimal, non-critical, assistance and four points if they were able to complete the case without any assistance from the investigator. EMN PTTNA skill was also divided into four domains and graded similarly: 1) pre-procedure evaluation and planning, 2) choice of insertion site, 3) needle insertion and advancement to target, 4) performance of biopsy. For both procedures, an individual domain/task score ≥ 3 (Table 1) and a total procedural score of ≥ 12 was considered “competent.”

TABLE 1.

Procedural Assessment Tool

		Required major assistance from investigators	Required minimal, but critical assistance from investigators (critical=input that significantly changed the diagnostic outcome or avoided potential complication)	Able to complete, but required minimal (non-critical) assistance from investigators	Able to complete without assistance, highest level of performance	Total
		One Point	Two Points	Three Points	Four Points
EMN Bronchoscopy	Item 1 Ability to plan procedure	□Identify appropriate indication for patient □Identify 6 sensors for planning □Find the lesion and select as a target □Review and approve airway plan □Grow airways if needed (identify bronchus sign) □Identify any inspiratory/expiratory target Motion
	Item 2 Equipment setup and registration/survey	□Correct vPad Placement □Field Generator Positioning □Accuracy Verification and Refinement if needed □Luminal Registration (If Applicable)
	Item 3 Navigate to target lesion	□Reach the correct segment with the bronchoscope prior to using navigation □Recognize airway movement (towards or away from lesion) □Navigate to the target and recognize when within 2cm of target
	Item 4 Perform biopsy	□Identify crosshairs within target on computed tomography planes (distance to target 0mm) □Identify biopsy instrument at target (set tip extension if needed) □Biopsy performed
EMN PTTNA	Item 1 Ability to plan procedure	□Identify appropriate indication for patient □Identify 6 sensors for planning □Find the lesion and select as a target □Review and approve airway plan □Identify any inspiratory/expiratory target motion
	Item 2 Choice of insertion site	□Identify target and entry location □Avoid crossing fissures or major vessels/organs
	Item 3 Needle insertion to target	□Place non-sterile needle on targeted body surface and identify appropriate trajectory □Setup appropriate supplies for needle biopsy □Place sterile needle without withdrawal from parenchyma □Recognize pathway and trajectory □Maintain needle pathway on targeting views □Direct need tip to target □Proper length and use of biopsy gun or needle
	Item 4 Perform biopsy	□Identify crosshairs within target on computed tomography planes (distance to target 0mm) □Identify biopsy instrument at target (set tip extension if needed) □Biopsy performed

After development of the above assessment tool, a round-robin discriminant validation was performed using three investigators in grading each of the other two during performance of EMN bronchoscopy and PTTNA.¹¹ The blinded scores were consistent between all of three investigators without inter-rater disagreement. All investigators scored 4’s (Mastery) in all domains of the assessment tool.

Recruitment

Subjects for the study were recruited at three separate courses for training in EMN bronchoscopy and PTTNA. The courses instructed by physicians with expertise in these procedures (HL, LY, SA). All subjects provided oral consent prior to participation in the study.

EMN Course

Each of the three courses were identical in format and consisted of a half day of didactic lectures using images/video and a second half of a hands-on simulation with cadavers. The investigators demonstrated EMN bronchoscopy and PTTNA to subjects prior to subjects performing the first simulation case.

Scoring of Subjects

After the didactic teaching and observation of a procedure, subjects performed biopsies of simulated synthetically created PPNs in cadavers via EMN bronchoscopy and PTTNA. Each subject was graded on every biopsy attempt using the previously established assessment tool. Feedback was given to subjects from the investigators based on the checklist after each attempt. The subjects repeated the procedures until reaching a competent score (≥3 in all four domains, total score ≥12) implying competency to perform the procedure without significant assistance from the investigator. The subjects’ scores and number of attempts to reach a competent score were recorded.

After completion of the didactic course an investigator (HL, LY, SA) was then present during the subject’s first live case performing an EMN bronchoscopy or PTTNA at their respective institutions. The investigator graded the subject at that time using the same checklist assessment tool and the subjects then self-graded themselves using the same assessment checklist. These scores were recorded along with complications and biopsy results for each live case. A global rating scale (GRS) was also recorded by the investigator for each first live case.

Data Collection

Upon agreeing to participate in the study, demographic information was obtained from each subject including: age, sex, hospital practice setting, postgraduate year level, sub-specialty training, number of years out of training, hand dominance, bronchoscopic experience over the past two years and prior EMN experience. The first simulation score, final simulation score, first live case score and self-assessment score for each of the four sub-sections as well as the total score was recorded. The procedure duration and whether the sample was diagnostic was also recorded and compared for both types of procedures.

Statistics

All categorical variables concerning competency were abridged to binary outcomes-competent or not competent, as the presence of competency was the primary goal of this study. Two-sample tests of proportions were used to compare the proportion of individuals competent before and after training, as well as to compare proportions of physicians competent after training to the actual procedure. In order to determine concordance between the observer and self-scores after the procedure, McNemar’s test was used. A correlation coefficient was also determined between the investigator scoring and subjects’ self-scoring for the first live case. All statistics were done using STATA IC, version 14.1.

RESULTS

Demographics

In total 14 physician subjects were recruited, all having completed fellowships in pulmonary medicine. Four physicians reported prior experience with a different EMN platform. Demographics and prior bronchoscopy data are described in Table 2.

TABLE 2.

Demographics and case description

	All	EMN Bronchoscopy	EMN PTTNA
Age (years), mean (median, range)	41.8 (40, 32–51)	41.1 (40, 32–51)	41.4 (40, 32–51)
Gender
Male	13	11	10
Female	1	1	0
Years out of Fellowship, mean (median, range)	7.7 (3, 1–20)	7.5 (3, 1–18)	7.7 (2, 1–20)
Bronchoscopy experience (in last 2 years)
40–60	2	1	1
120–140	5	4	3
>150	7	7	6
Practice
Community		8	6
Academic		4	4
Duration (minutes) (mean, median, range)		38.7 (38, 20–51)	22 (21, 14–30)
Days Between Simulation and First Live Case, mean (median, range)		44.5 (45, 5–77)	58 (72.5, 6–81)

EMN Bronchoscopy

The mean first simulation score for EMN bronchoscopy after classroom didactic teaching/observation was 4.67 ± 0.88. After additional hands-on teaching using the cadaver simulators, subjects improved to a final simulation score of 12.6 ± 0.88 (pre-set competent score ≥12.0). The mean and median number of attempts to reach a competency score were 3 attempts. The mean score for their first live case was 13 ± 0.85 (Fig 1), significantly better than the first simulation score (P =0.0005) and not significantly different from the final simulation score (Kappa-100%, p=1.0). The subjects’ self-reported mean score was 12.58 ± 0.90. There was no significant difference between the subjects’ self-reported score and the first live case score (R=0.829, p=1.0).

Figure 1.

Total procedural assessment scores.

EMN PTTNA

For EMN PTTNA, the mean first simulation score was 4.3 ± 2.40 with subjects improving to a final simulation score of 12.6 ± 1.5. The mean and median number of attempts to reach a competency score was 3 attempts. The subjects’ first live EMN PTTNA case had a mean score of 12.5 ± 2.87 (Fig 1). This was significantly improved from their first simulation score (P=<0.001) and similar to their final simulation score (p=0.3049). The correlation between the final simulation score and the subjects’ self-reported score was strong (R=0.959, p=1.0). The subjects’ self-reported score was 12 ± 0.15. There was no significant difference between the subjects’ self-reported score and the first live case score. Only one subject failed to accomplish a competent score in the first live case. This subject had gone on to remediation with additional training and had a competent score (not included in the analysis) on the next clinical procedure

Global Rating Scale Evaluation

The overall GRS scores for EMN bronchoscopy and PTTNA were not significantly different from the assessment scores and had a perfect correlation for competency (100% correlation, p=1.0).

Complications

There was one minor complication related to bleeding immediately after EMN PTTNA which was managed with bronchoscopy suction and epinephrine without hemodynamic changes or need for hospitalization.

DISCUSSION

This study examined the procedural learning process using an assessment tool, simulation training, and structured feedback. Our educational construct was based on social learning theories of observation (modeling) and repetition/reinforcement in different environments.¹² After didactic learning/observation alone, no subjects reached our defined competency on their first simulation attempt. This was despite all subjects having background experience with bronchoscopy and possibly other translatable skill sets. This challenges the old adage of “see one, do one, teach one.” Only through repetition coupled with post-simulation feedback were subjects able to reach competency in all the measured domains. This is consistent with the Dreyfus model of skill acquisition which has been described for surgical skills acquisition. Most importantly, the subjects’ competency remained consistent between the final simulation procedures into their first live patient procedures for both the EMN bronchoscopy and PTTNA. Our study demonstrated the translation of skills from simulation into a “real world” scenario, reflecting the true apex of Miller’s pyramid.¹³ This is a common misstep in most simulations studies in not demonstrating the transference of skills from simulation into clinical practice. This proficiency was independent of investigator scoring and supported by the safety outcomes as well.

The structured feedback/reinforcement based on the assessment tool allowed for reproducibility and “shaping” through positive/negative feedback. While our assessment tool was based on an already validated tool, we had modified the rubric of the tool to fit our procedures and remains validated by several measures in this study. Content validation in this study was established through expert investigators of a novel procedure. In addition, discriminant and convergent validation was demonstrated in a similar fashion to previous studies such as the EBUS-STAT and rigid task.^{3, 14} Our clinical discriminant validation is further supported by demonstrating improvement from the previously mentioned “non-competent” scores to the competent range after simulation training. Clinical validity was established in the blinded round-robin procedural scoring of investigators, where all scores were expectedly in the mastery range. At the opposite spectrum, first simulation scores for all study subjects, who were novices for both procedures, were in the non-competent range.

Several studies have suggested the higher sensitivity of a GRS system compared to a checklist assessment tool for assessing procedural competency.^154–18 In our study, there was a strong correlation between our assessment tool and the GRS, which also evaluated each first live case. While prior studies have suggested advantages of a GRS over a checklist as an assessment tool, the checklist system may provide additional educational value beyond competency metrics by formalizing structured feedback around the content of the checklist. This is an important notion as our current bronchoscopy assessment tools are based on a checklist system and suffers from the same shortcomings. In designing our assessment system, we had gone beyond the traditional checklist system (“yes or no”) which has been criticized by various experts because of its rigidity¹⁵. Rather than a “yes or no” check box, we included some subjectivity similar to a GRS system which also allows for the proctor to input subjectivity based on the situation at hand. Also, unlike prior published assessment tools for bronchoscopy, our assessment tool was not in isolation but rather incorporated into a simulation course which is more likely to represent real world practices when teaching procedures. The division of our assessment tool into four separate domains also allows for a subjective acceptable cognitive load for learners/trainers.

Only one subject was not able to demonstrate a competent score for EMN PTTNA despite having a competent score on the final simulation case. This stresses the importance of objective testing after simulation training in a live patient environment. There can be numerous factors (i.e. extinguishing of learned skills, performance anxiety) that may alter performance from a simulated environment into high stakes live clinical cases In this single case, after additional training, the subject was able to perform competently on their next live case. In the United States, procedural competency is evaluated during fellowship/residency by their program directors who are best positioned to assess their abilities. However, after formal training, competency is often decided by credentialing boards at institutions which may not include physicians in the same specialty or use proctorship style systems to evaluate competency. With growing public expectations about quality and transparency in health care delivery, our study may provide a potential model for assessing competency after residency/fellowship.

Our current graduating physicians will likely have to learn new procedural skills during their career unlike prior generations where skills learned during fellowship may last throughout their careers. The current system was also designed to be adaptable and replicated for future procedural technologies as they become available to asses competency. An expert panel may need to list the critical aspects of the procedural checklist as we did in our study, but can keep the current scoring system and training methodology (assessment at various time points). We in essence demonstrated the feasibility of using the same scoring/training methods for two different procedures; PTTNA and EMN bronchoscopy. As validation is a continuous process, prior to such an adoption, additional investigations should be conducted on the inter-testing reliability between different observers and their skill levels. It may require instruction for observers to understand prior to its use. It is also worth to note that this assessment tool has only been validated for learners of this particular program and will need to be studies further for different learning formats and students (fellows). As more technological procedures become available to us, it is unrealistic for educators to master a competency tool for each procedure (i.e. EBUS Stat, Rigid Task, LEAP) rather a single metric system may be used with different rubrics.

When examining self-reported scores of subjects compared to the first live case scores, there were no significant differences. This may be influenced by a study environment and it remains unclear if a self-assessment alone would be sufficient to establish competency. However, none of the subjects reported a higher score than the investigators.

The authors do recognize the limitations of this pilot study, including the lack of a formal control group. It was felt to be unethical to have a non-simulation training group given the extent of literature supporting simulation procedural training. Rather we used each subject as their own control by comparing the change in scores during training exposure. Also, there was no follow up assessment after their first live case, as maintenance of skill proficiency was outside the scope of this pilot study. A larger study with multiple evaluation points should be the next step to validate our findings. Additionally, the investigators may introduce a bias by performing the assessment tests in this study. Future studies would require multiple independent evaluators in assessing this method. As EMN PTTNA was a completely novel procedure for all of these physicians, this is less likely to be the case. We opted to use a high fidelity simulator using cadavers to more accurately replicate airway anatomy/tactile resistance of trans-thoracic needle puncture. The availability and cost of cadavers in the United States differs vastly and may limit the replication of this form of training. It is also unclear if low fidelity simulators would be able to accomplish the same results as other investigators have demonstrated for other procedures.

CONCLUSION

Competency for EMN bronchoscopy and PTTNA is feasible using a structured simulation course with a validated competency assessment tool.

ABBREVIATIONS

EMN

electromagnetic navigational

GRS

global rating scale

PPN

peripheral pulmonary nodule

PTTNA

percutaneous transthoracic needle aspiration