Using structured progress to measure competence in flexible bronchoscopy

Abstract

Background

Flexible bronchoscopy is a core invasive procedure in pulmonary medicine and training in the procedure is mandatory. Diagnostic completeness and procedure time have been identified as useful measures of competence. No outcome measures have been developed regarding navigational path in bronchoscopy to assess whether the bronchial segments have been identified in an arbitrary or structured order. We investigated whether a new outcome measure for structured progression could be used to assess competency in flexible bronchoscopy.

Methods

The study was designed as a prospective comparative study. Twelve novices, eleven intermediates, and ten expert bronchoscopy operators completed three full bronchoscopies in a simulated setting on a phantom. The following outcome measures were collected through a checklist evaluation by a trained rater: Diagnostic Completeness as amount of visualized bronchial segments, Structured Progress between the bronchial segments in ascending order, and average intersegmental time (AIT).

Results

The ability to follow a structured ascending path through the bronchial tree correlated with a higher amount of identified bronchial segments (Pearson’s correlation, r=0.62, P<0.001) and a lower AIT (Pearson’s correlation, r=−0.52, P<0.001).

Conclusions

Operators should advance through the bronchial tree in a structured ascending order to ensure systematic progress with the highest level of diagnostic yield and the lowest procedure time. Structured progression is a useful measure to evaluate competency in flexible bronchoscopy.

Introduction

Flexible bronchoscopy is an essential part of the diagnostic workup of patients suspected of lung cancer (1). Correct procedural performance is a prerequisite for accurate diagnosis and allocation of correct treatment of the patient (2). A lower yield of positive biopsy material, increased patient discomfort, and higher complication rates are seen in the early part of a trainee’s learning curve (3-5) and a satisfactory educational level must be ensured before independent/unsupervised practice on patients. Traditionally, certification has been based on an arbitrary number of procedures performed as suggested by The American College of Chest Physicians (100 procedures) (6), and the European Respiratory Society (7). These suggestions are not evidence-based since trainees learn at different paces and procedural experience does not ensure competency (8,9). A CHEST expert panel report suggested a move from volume-based certification system to skill acquisition and knowledge-based competency assessment for trainees (10). Mastery learning is a training modality, where the trainee practices until proficiency targets are met (11), and thereby relies on assessment tools that can determine if the desired level of competence is achieved (12). Several tools have been developed for the assessment of bronchoscopy performance (13,14). These assessment tools allow for the evaluation of diagnostic completeness and procedure time which have been identified as useful measures of competence when performing flexible bronchoscopy (14-17). However, there are no measures which examine whether the bronchial segments are inspected in a random or structured order. A fundamental element in performing bronchoscopy is that the operator should be able to identify all segments without overlooking any of these, since each segment may contain a tumor or other important pathology even in a patient not suspected of lung cancer (18). We hypothesize that following a systematic ascending path through the bronchial tree ensures a higher fraction of examined bronchial segments and limits the procedure time, and therefore can be regarded as an assessment measure of operator competency.

The aim of the study was to explore whether a new outcome measure for structured progression through the bronchial tree could be used to evaluate competency in flexible bronchoscopy.

We present the following article in accordance with the STROBE reporting checklist (available at http://dx.doi.org/10.21037/jtd-20-2181).

Methods

Study design

The study was designed as a prospective comparative study performed in a simulated setting at the Copenhagen Academy for Medical Education and Simulation. Messick’s framework of validity was used to gather validity evidence from four sources: content, response process, internal structure and relations to other variables (Table 1) (19). It is the gold standard when evaluating validity as recommended by the American Educational Research Association (20,21).

Table 1. Different sources of validity evidence for Structured Progress based on Messick’s validity framework (19).

Source of evidence for validity	Description of source	Validity evidence for structured progress
Content	The test content measure what it is supposed to measure	Structured progress (SP) was developed by two pulmonary consultants (PFC and UB) with more than 10 years’ experience in bronchoscopy, a thoracic surgeon and professor of medical education (LK) and a biomedical engineer with more than 5 years’ experience in simulation (MBSS). SP indicates the number of times the operator progressed from one segment to the immediate succeeding segment. Passing from segment #1 to #2 gives one point but going from segment #2 to #1 gives zero point. It ranges from 0–18 points, corresponding to 18 segments in the bronchial three.
Response process	Integrity of data should always be maintained. Test administration should be controlled or standardized at a maximum level possible.	All trials were performed in a controlled, simulated environment, each rating was blinded and double–checked with the video recording by the primary investigator KMC.
Internal structure	This refers to the reliability of the test results. The internal consistency of the test items should result to similar scores when measuring the same construct.	There was a correlation between all three outcome measures: Structured progress, diagnostic completeness, and AIT both across and within different groups of experience. Thereby, test items result to similar scores when measuring competency in bronchoscopy.
Relation to other variables	Assessment scores should correlate with known measures of competence	Structured progress was able to significantly differentiate the performances among the different levels of experience with median scores ± interquartile range for novices, intermediates and experienced of 5±3, 7±4.75 and 11±5, respectively (P<0.001)

Simulation equipment

The simulated set-up included an endoscopy tower with a flexible bronchoscope (EVIS Exera II and Q180 flexible bronchoscope, Olympus, Japan). The phantom was a bronchoscopy model equipped with a three-dimensional bronchial tree (CLA Broncho Boy, CLA, Coburg, Germany). The right lung has ten segments (segments #1–#3 in the upper lobe; #4 and #5 in the middle lobe; #6–#10 in the lower lobe), while the left lung has eight segments (#1–#3 in the upper lobe with #1–#2 fused together to form one segment; #4–#5 in lingula; #6–#10 in the lower lobe without a segment #7).

Performance checklist

A checklist that recorded specific checkpoints corresponding to each bronchial segment of the phantom was developed for the purpose of the study. This was set up in a graphical user interface written in Python [3.7.6] using the Tkinter library included in the standard package.

Participants

The participants were recruited and divided into three groups based on levels of experience: The novice group included physicians with no prior experience in flexible bronchoscopy, the intermediate group included physicians who had performed 5–100 procedures, and the expert group included physicians with experience in >500 procedures. For the study size each group should contain at least ten participants. The inclusion period was from 1st of March to 31st of May 2019.

Ethics

Participation was voluntary; informed consent was gathered before inclusion. In accordance with Danish law, an educational study without participation of patients does not require ethical approval.

Assessment procedure

The participants were invited to participate in the study through email. The purpose of the study and the included parameters were not disclosed to the participants in order not to influence their performance. Prior to coming to the simulation center for testing, they received a basic handbook in flexible bronchoscopy to use as a reference, which included a description of the anatomy of the bronchial tree and how to systematically locate all bronchial segments (22). This handbook included a description of the anatomy of the bronchial tree and how to find all bronchial segments.

Testing was performed individually. Each participant initially practiced using the flexible bronchoscope by locating four hidden marks inside a makeshift box. Afterwards, they were asked to perform three bronchoscopy procedures on the phantom.

During the bronchoscopy, the operators could not see the outside of the tracheobronchial tree inside the chest (Figure 1). They were asked to enter all segmental bronchi and were not given any feedback during or in-between the trials.

Figure 1.

Live rating using a checklist of bronchial segments while the participants performs a full bronchoscopy on an inanimate bronchoscopy model, Broncho Boy (CLA, Germany).

Rating

The procedures were witnessed and monitored by the primary investigator (KMC) who has extensive experience in performing and rating bronchoscopy in a simulated setting. Each participant was rated during the procedure using the checklist. When the button representing a bronchial segment was pressed, the time and position of the scope were saved automatically in a text file.

All recordings were blinded and verified for the correct amount of visited bronchi by KMC.

Outcome measures

A new outcome measure, Structured Progress (SP) was developed by two pulmonary consultants (PFC and UB) with more than 10 years’ experience in bronchoscopy, a thoracic surgeon and professor of medical education (LK) and a biomedical engineer with more than 5 years’ experience in simulation (MBSS). SP indicates the number of times the operator progressed from one segment to the immediate succeeding segment. Passing from segment #1 to #2 gives one point but going from segment #2 to #1 gives zero point. SP ranges from 0–18 as the potential first point is awarded by passing from the carina to segment #1 either on the right or left side depending on which lung is inspected first. Progression from the carina to segment #1 to segment #2 to segment #3 gives 3 out of three points, while progressing from the carina to segment #1 to segment #3 to segment #2 gives 1 out of three points.

Other measures include diagnostic completeness (DC) defined as the thoroughness of examination measured as the fraction of total bronchial segments entered as a total of 18 segments and average intersegmental time (AIT), which is the average time passed between each segmental visit, calculated based on the difference between timepoints registered.

These three outcome measures were automatically calculated by the graphical interface after each performance.

Data analysis and statistics

The performances of the three groups with regards to SP, DC, and AIT were collected through the checklist software and compared using Kruskal-Wallis with a post hoc Dunn-Bonferroni test. The correlations between SP, DC, and AIT were calculated as Pearson’s r value.

The three groups were compared according to sex, age, medical specialty, individual and supervised bronchoscopies in a clinical setting, bronchoscopies in a simulated setting, bronchoscopies within the last 12 months and days since the last bronchoscopy by ANOVA for data showing normal distribution or by Kruskal-Wallis for data that did not show normal distribution for continuous variables, and by χ²-test for categorical variables.

For data with normal distribution, mean and standard deviation were used. For data that did not show normal distribution, median and range were used.

All data analysis was completed in Python (3.6.6, Python Software Foundation), using libraries Numpy (23), Scipy (24), and Matplotlib (25). Statistical testing was completed in Statistical Package for the Social Sciences version 25 (PASW v25.0; SPSS Inc., Chicago, Ill., USA). A P <0.05 was considered statistically significant.

Results

A total of 33 participants were enrolled: 12 novices, 11 intermediates, and 10 experts (Table 2). All participants performed three procedures each resulting in 98 trials. The first trial data of one of the intermediates was lost due to system breakdown.

Table 2. Participants details and comparison.

Variable	Novice (n=12)	Intermediate (n=11)	Expert (n=10)	P
Male gender, number (percent)	8 (0.67)	6 (0.55)	7 (0.70)	0.735a
Age mean ± SD, years	34±5.1	38±5.7	49±5.0	<0.001b
Individual bronchoscopies in a clinical setting, number	0.0±0	10.0±25.0	1,000.0±1,650.0	<0.001c
Supervised bronchoscopies in a clinical setting, number	0.0±0	40.0±67.0	45.0±97.0	<0.001c
Simulated bronchoscopies, number	0±0	15.0±20.0	7.5±11.0	<0.001c
Bronchoscopies in a clinical setting within the last 12 months, number	0±0	5.0±45.0	200.0±118.0	<0.001c
Days since last performance of a bronchoscopy, days	–	29.0±999.0	2.5±5.25	<0.001d

Values are presented as median ± interquartile range or as otherwise indicated. P was compared by Kruskal-Wallis, ANOVA, student’s independent t-test and Mann-Whitney U test for continuous variables or by χ²-test for categorical variables. ^a, the result of the comparison between the three groups by a χ²-test; ^b, the result of the comparison between the three groups by a one-way ANOVA; ^c, the result of the comparison between the three groups by a Kruskal-Wallis test; ^d, the result of the comparison between the two groups by a Mann-Whitney U test.

A summary of the results based on validity evidence for the four sources are presented in Table 1.

Content

Structured progress is a progressive score ranging from 0–18, indicating the number of times the operator progressed from one segment to the immediate succeeding segment.

Relations to other variables

All three outcome measures (SP, DC, and AIT) were significantly different between the groups using Kruskal-Wallis (P<0.001) (Table 3, Figure 2). For the expert group, DC was not correlated with individual bronchoscopies in a clinical setting (Pearson correlation, r=0.23, P=0.42), bronchoscopies in a simulated-setting (Pearson correlation, r=0.50, P=0.14), bronchoscopies within the last 12 months (Pearson correlation, r=−0.25, P=0.48) or days since last bronchoscopy (Pearson correlation, r=0.21, P=0.56).

Table 3. Trial details and comparison.

Group/variable	Novice trials (n=36)	Intermediate trials (n=32)	Expert trials (n=30)	Pa	Post-hocb
SP, number	5.00±3.00	7.00±4.75	11.00±5.00	<0.001	A, B, C
DC, fraction	0.83±0.17	0.89±0.17	0.94 ±0.07	<0.001	A, C
AIT, seconds	34.91±20.99	14.22±6.01	11.74±7.21	<0.001	A, C

^a, the result of the comparison between the three groups by a Kruskal-Wallis test; ^b, the result of the pairwise group comparison by a Dunn-Bonferroni post-hoc test. Values are presented as median ± interquartile range. P values were calculated using Kruskal-Wallis test with Dunn-Bonferroni post-hoc test. Number (n) indicates amount of trials. One trial data of intermediate 6 was lost due to system breakdown. All other trials were fully recorded and analyzed. Post-hoc A indicates statistical significance between novice and intermediates. Post-hoc B indicates statistical significance between intermediates and experts. Post-hoc C indicates statistical significance between novice and experts. AIT, average intersegmental time. Average time passed between each segmental visit. DC, diagnostic completeness. Number of visualized segments divided by total number of segments. SP, structured progress. A score from 0–18 points. One point was given every time the operator proceeded from one segment to the immediate succeeding segment.

Figure 2.

Left: DC as a fraction according to the score of SP in points from 0 to 18. Center: AIT in seconds according to the score of SP in points 0 to 18. Right: AIT in seconds according to the score of DC as a fraction. For all three figures bards indicate reference interval of each group. DC, diagnostic completeness; SP, structured progress; AIT, average intersegmental time.

Internal structure

There was a significant correlation between all three outcome measures (Table 4): SP and DC (Pearson correlation, r=0.62, P<0.001), SP and AIT (Pearson correlation, r=−0.52, P<0.001) and DC and AIT (Pearson correlation, r=−0.52, P<0.001). Sub-analysis of each group showed moderate correlation for all three measures except for the novice group at the correlation between SP and AIT (Pearson correlation, r=−0.22, P=0.20) (Table 4).

Table 4. Correlation between outcome variables.

Group/variable	Novice trials (n=36)	Intermediate trials (n=32)	Expert trials (n=30)	Total trials (n=98)
SP vs. DC	0.51 vs. 0.01	0.45 vs. 0.01	0.63 vs. <0.001	0.62 vs. <0.001
SP vs. AIT	−0.22 vs. 0.20	−0.54 vs. <0.001	−0.51 vs. 0.01	−0.52 vs. <0.001
DC vs. AIT	−0.376 vs. 0.02	−0.63 vs. <0.001	−0.56 vs. <0.001	−0.52 vs. <0.001

Values are presented as Pearson’s correlation coefficient r and P. P was compared as Pearson’s correlation. Number (n) indicates amount of trials. One trial data of intermediate 6 was lost due to system breakdown. All other trials were fully recorded and analyzed. AIT, average intersegmental time. Average time passed between each segmental visit. DC, diagnostic completeness. Number of visualized segments divided by total number of segments. SP, structured progress. A score from 0–18 points. One point was given every time the operator proceeded from one segment to the immediate succeeding segment.

Discussion

This is the first study to explore the importance of a structured navigational path in bronchoscopy. We found that following a structured navigational path in ascending order between the bronchial segments (SP) was correlated with a higher DC and a lower AIT.

More specific guidelines that govern training in bronchoscopy are needed (26). Previous studies in simulation-based training in bronchoscopy have proposed DC as a measure of competence but did not give any suggestions towards a training method to improve this measure (17,27). DC was not attributable to recent or total experience with the procedure for the expert group. This finding is in line with the conclusion that the amount of performed bronchoscopies act as a surrogate marker for competence when assessing bronchoscopy skills (28). Our study shows that performance of a certain number of bronchoscopies does not ensure competency in reaching all segments. Based on our findings we suggest that following a structured navigational path of the procedure could instead be used as an indicator for competence.

Our study showed a positive correlation between DC and SP (Pearson correlation, r=0.62, P<0.001). Several findings in our study indicated this correlation to be a causal relation. It seems obvious that operators should visualize as much of the bronchial tree as possible; to ensure this, a bronchoscopy should optimally be completed in a systematic approach. This finding holds an obvious confounder, as the groups were divided by experience level. When analyzing only the expert group there was no significant correlation between DC and experience but the correlation between DC and SP was still significant (Pearson correlation, r=0.63, P<0.001), indicating that the most systematic experts managed to visualize more of the bronchial tree than their colleagues.

The segments are named after an intended orderly progression in bronchoscopy (29). Yet our study shows that the systematic approach is absent to a high degree when observed in a simulation setting. It has therefore been proposed for decades that bronchoscopy should be navigated through a structured progression between the bronchial segments (29). This study is the first to provide evidence for this assumption.

It is crucial to be systematic when performing an endoscopic pulmonary procedure (27). This has been shown for endobronchial ultrasound guided biopsy (EBUS-TBNA) for the staging of lung cancer. When comparing a systematic approach to a targeted, the former resulted in a higher diagnostic yield (30). Inspection of all segments is important as bronchoscopy can reveal endobronchial tumors that are not visible with computer tomography (31) and in patients not suspected for lung cancer (18). Similar results have been found in the field of colonoscopy (32,33). Therefore, a systematic approach is important when performing flexible bronchoscopy to ensure no segments are missed, and our measure of SP can be used as a valid assessment tool for user competence. The scoring system for SP is also adaptable for example in the case of a known abnormality where bronchoscopic inspection should start in the contralateral side of the lesion. One point is awarded for each systematic progression such that if the inspection starts in the left lung, the first point is given when progressing from carina to segment #1 on the left side.

A systematic progression in this study was associated with lower procedure time, as SP correlated with a decreasing AIT (Pearson correlation, r=−0.52, P<0.001). Procedure time has been identified as a measure of competence (14,15). Reducing the procedure time is an important aspect of bronchoscopy as a time effective procedure limits the amount of patient sedation, discomfort, and costs (34,35).

Simulation continues to play an important role for skills assessment (36). Virtual reality simulators combined with expert raters are currently used to asses competence in bronchoscopy (13). However, the use of a phantom can reliably be used to assess skill level (37) and could be superior to virtual reality simulators (26,38). The disadvantage of training on a phantom is that they do not provide any direct feedback as compared to virtual reality simulators. Computerized feedback improved the performance of trainee endoscopists as compared to no feedback (39).

Based on this study we aim in the future to develop a computerized feedback system that is feasible when training on phantoms to ensure a higher learning outcome. This could also work as an automatic assessment tool, which will be able to set an unbiased pass/fail criterion that is not susceptible to rater bias. Future studies should compare our automatic, objective scores with existing more subjective scoring systems from expert raters (40).

Our study is the first to examine the importance of a structured navigational path for flexible bronchoscopy and has several limitations. It was a single-center study, but participants were from 11 different hospitals in Denmark. Our sample size was relatively small but comparable to average educational studies (41). All data collection was done in a simulated setting to ensure full standardization and avoid ethical concerns. We used a phantom which is static, with no possibilities to train complexities and only exposes the learner to one iteration of the bronchial three. However, this is a first step towards developing an assessment tool of navigational path and it would be useful to repeat the study in real patients and in complicated bronchoscopy procedures. The study only focused on a limited amount of outcome measures. DC and AIT have been identified as useful measures of competence when performing flexible bronchoscopy (14,15,17).

This study identified SP as a useful outcome measure to assess competency in bronchoscopy. SP can be used alongside existing measures of competence (DC and AIT) (14,15,17), adding new useful nuances to the assessment of bronchoscopy performance that are not captured by existing measures—it was the only of the three outcome measures that was able to differentiate between intermediates and experts. We aim in the future to further develop the assessment tool to generate automatic feedback regarding the outcome measures DC, SP and AIT and to set predefined pass/fail standards for competent performance.

Supplementary

The article’s supplementary files as

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. Participation was voluntary; informed consent was gathered before inclusion. In accordance with Danish law, an educational study without participation of patients does not require ethical approval.