Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2017 Aug 17.
Published in final edited form as: IEEE Trans Biomed Eng. 2015 Feb 6;62(12):2794–2811. doi: 10.1109/TBME.2015.2401514

Hands-Free System for Bronchoscopy Planning and Guidance

Rahul Khare 1, Rebecca Bascom 2, William E Higgins 3,
PMCID: PMC5560505  NIHMSID: NIHMS879557  PMID: 25675452

Abstract

Bronchoscopy is a commonly usedminimally invasive procedure for lung-cancer staging. In standard practice, however, physicians differ greatly in their levels of performance. To address this concern, image-guided intervention (IGI) systems have been devised to improve procedure success. Current IGI bronchoscopy systems based on virtual bronchoscopic navigation (VBN), however, require involvement from the attending technician. This lessens physician control and hinders the overall acceptance of such systems. We propose a hands-free VBN system for planning and guiding bronchoscopy. The system introduces two major contributions. First, it incorporates a new procedure-planning method that automatically computes airway navigation plans conforming to the physician’s bronchoscopy training and manual dexterity. Second, it incorporates a guidance strategy for bronchoscope navigation that enables user-friendly system control via a foot switch, coupled with a novel position-verification mechanism. Phantom studies verified that the system enables smooth operation under physician control, while also enabling faster navigation than an existing technician-assisted VBN system. In a clinical human study, we noted a 97% bronchoscopy navigation success rate, in line with existing VBN systems, and a mean guidance time per diagnostic site= 52 s. This represents a guidance time often nearly 3 min faster per diagnostic site than guidance times reported for other technician-assisted VBN systems. Finally, an ergonomic study further asserts the system’s acceptability to the physician and long-term potential.

Index Terms: Bronchoscopy, computed tomography, image-guided intervention systems, lung cancer, procedure planning, surgical guidance, virtual bronchoscopy (VB), 3D imaging

I. Introduction

Bronchoscopy is a commonly used minimally invasive procedure for lung-cancer staging [1]–[3]. To perform bronchoscopy, the physician first inspects a patient’s high-resolution 3-D computed-tomography (CT) chest scan to identify diagnostic regions of interest (ROIs) and reconstruct airway routes leading to the ROIs [2], [4]. During live bronchoscopy, the physician uses the CT-derived procedure plan to navigate the bronchoscope to each ROI by registering the bronchoscope’s video stream to the CT-based route plan, with radiographic fluoroscopy often used to help localize final diagnostic sites [2], [3]. It is well known, however, that physicians not only differ substantially in their ability to perform bronchoscopy, but also perform poorly in general for diagnostic sites situated in more peripheral lung regions [5], [6]. As a response to these issues, researchers have proposed image-guided intervention (IGI) systems for bronchoscopy [6]–[26]. IGI systems have been shown to greatly increase bronchoscopy success rate, while also largely eradicating the skill variations between physicians [7], [8].

To achieve this, IGI systems draw upon the notions of a “virtual” airway-tree model and virtual bronchoscopy (VB), both derived from a patient’s chest CT scan [27]–[32]. In the 1990s, researchers noted that VB views—i.e., CT-derived renderings of the virtual airway-tree interior—strongly resemble a bronchoscope’s video views of the “real” airway-tree interior [27]–[30]. Using this observation, the bronchoscope’s position within the airway tree can be established by synchronizing VB views at known locations along a preplanned airway-tree route to video frames obtained from the bronchoscope. The class of IGI systems using this image-to-image synchronization approach are referred to as virtual bronchoscopic navigation (VBN) systems [11], [12]. Bronchoscope positioning can also be accomplished by registering the 3-D location of a bronchoscope sensor immersed in an electromagnetic field to the CT-derived virtual chest space [18]. The class of IGI systems using this sensor-to-image synchronization approach are referred to as electromagnetic navigation bronchoscopy (ENB) systems [20], [21].

A meta-analysis of 39 recent clinical studies using IGI systems for bronchoscopy found that VBN systems [8]–[17] and ENB systems [18]–[22] offer a comparable level of performance. Investigators, however, have noted several disadvantages of ENB systems [12], [21]. First, differences between the static CT-based virtual anatomy and patient’s “live” dynamic anatomy, which constantly changes due to respiration, result in registration errors—such errors can be especially large in the peripheral lung lobes. Second, for systems using a sensor not attached to the bronchoscope, respiratory movements can dislodge the sensor, making reliable biopsy-site selection problematic. These two issues can result in pneumothoraces and low biopsy yield. As a third issue, the learning curve for EBN systems can be substantial. Fourth, ENB systems do not typically provide direct endoluminal video views of the airway tree interior during bronchoscope navigation. Instead, they display VB views nominally corresponding to the current registered position; this implies that navigation is blind. Fifth and most significant, these systems entail a considerable extra cost for the sensor and tracking hardware and also for the specialized disposables typically required for each patient.

VBN systems, on the other hand, are not susceptible to the disadvantages attributed to ENB systems [8]–[17]. They do not require expensive sensor/tracking hardware or special disposables. Because navigation draws upon local synchronization between the bronchoscope’s video stream and CT-based VB views, VBN systems are far less susceptible to registration/positioning errors. In addition, since VBN systems display both the bronchoscopic video and VB views during guidance, they offer direct airway visualization at all times. This in turn makes it easy and intuitive to learn system operation. In fact, many of these systems enable the navigation routes and preselected ROIs to be superimposed onto the live video, enhancing the visualization experience. Given these attributes, our work focuses on VBN systems.

Before continuing, it is important to discuss a few aspects of standard bronchoscopy practice. First, pulmonary physicians undergo a standardized training regimen for bronchoscopy. In particular, they learn the natural rotate-flex-advance process for bronchoscope navigation [33], [34]. Second, clinical bronchoscopy is a team effort: the pulmonary physician performs the actual procedure, while a technician attends to the patient, sets up the bronchoscope hardware, manages the bronchoscopy-based tissue-sampling devices (e.g., needles and forceps), and handles cytology slides. Because bronchoscopy involves highly stressful decisions, the physician necessarily controls the pace and execution of the procedure, without undue interference from the technician.

Unfortunately, the physician’s ability to operate an IGI system is limited, since he/she generally needs both hands to manipulate the bronchoscope. While this limitation does not affect ENB systems, which can be operated autonomously by the physician, it does affect VBN systems: all existing VBN systems require the attending technician to co-pilot the system controls during a procedure [12], [15]–[17]. Furthermore, the physician must actively direct the technician as to when to perform every VBN guidance operation, e.g., when to advance the guidance system’s display to the next airway on a preplanned route. This extra distraction divides the physician’s attention, as the maneuvers involved can be complex [12], [16]. Thus, experience and coordination are essential for the clinical team to be able to work together smoothly. Without this, procedural errors can occur. (Asano discusses this issue in the review article [12].) Unfortunately, a VBN system’s need for technician assistance in addition to the added burden it places on the physician has hindered the acceptance of these systems. Nevertheless, as noted earlier, VBN systems have many factors in their favor over ENB systems for guiding bronchoscopy.

Two other limitations of all current IGI systems are also worth noting. First, while such systems usually offer automatic procedure planning, the resulting procedure plans do not translate into the natural rotate-flex-advance process learned by physicians during their training, i.e., the airway routes presented by the IGI systems during bronchoscopy could entail awkward device maneuvers. Second, no IGI system offers a position-verification mechanism in the case of a loss of visualization or abrupt movement (other than starting over), due to adverse events, such as patient cough, airway collapse, or excessive breathing motion.

We propose a VBN system that addresses the three aforementioned limitations. To do this, our system introduces two complementary contributions. First, it incorporates a new planning method that automatically computes airway navigation plans conforming to the physician’s bronchoscopy training. In particular, drawing upon the patient’s chest CT scan, the method derives navigation plans accounting for: 1) the physician’s hand position and dexterity; 2) airway-tree geometry; and 3) bronchoscope position. Each navigation plan consists of a predefined step-by-step sequence of rotate-flex-advance maneuvers that enables natural bronchoscope navigation. In this way, our VBN system provides natural comfortable guidance information abiding by the physician’s training. Our second contribution, which is tailored to exploit the navigation plan’s precomputed bronchoscope maneuvers, involves a hands-free method for bronchoscopy navigation that enables the physician to directly control the VBN system with a three-position foot switch—this eliminates the involvement of the technician. The system is driven by a new system-level guidance strategy that translates the navigation plan into instructions suitable for straightforward autonomous navigation. The guidance strategy also provides, for the first time, a position-verification mechanism that can indicate navigation errors and suggest corrective action.

As our studies presented later show, our VBN system enables substantially faster bronchoscopy navigation than existing VBN systems, without compromising bronchoscopy navigation success. In addition, an ergonomic study suggests that our system provides guidance information that satisfies the physician’s expectations. We believe that these factors, along with the mitigation of technician assistance, greatly improves the potential acceptability of the system in the long run. Overall, our VBN system requires less operational overhead than current VBN systems, while providing greater ergonomic acceptability and effective bronchoscopy to the lung periphery.

Section II describes the system and internal methods. Section III presents results from controlled phantom studies, a live patient study, and the aforementioned ergonomic study. Finally, Sections IV and V discuss observations noted in the studies and offer concluding comments.

II. Methods

Our VBN system consists of a guidance computer, monitor, keyboard, and mouse, all mounted on a mobile cart. The computer is a mid-range Dell T5400 64-bit workstation with a 3-GHz quad-core Xeon processor (16-GB RAM, 768M NVidia graphics card). The computer draws upon a Matrox Solios eA frame grabber for video stream interfacing. We developed all software in Visual Studio 2008 using C++. To enable hands-free operation, the system includes a programmable Delcom three-position foot switch (PN: 903603-5M), which interfaces to the guidance computer through a USB cable.

Application of our system to a patient procedure parallels the standard two-stage work flow of other IGI bronchoscopy systems, as summarized below [12], [16], [21].

  1. Procedure Planning

    Completed offline prior to the live procedure, this stage involves two steps:

    1. Three-dimensional CT analysis: Select desired diagnostic ROI sites, such as lymph nodes and nodules, in the patient’s chest CT scan. Next, perform automated analysis to define a 3-D airway-tree model and compute optimal airway routes leading to each ROI.

    2. Navigation plan computation: Starting with each target ROI’s associated airway route, automatically derive a navigation plan that consists of a series of natural bronchoscope maneuvers to follow during live bronchoscopy.

  2. Image-Guided Bronchoscopy

    Performed in the bronchoscopy suite after patient preparation, two steps are involved.

    1. System initialization: Interface the guidance computer to the bronchoscopy hardware and preview the navigation plan.

    2. Live bronchoscopy: The physician navigates the bronchoscope toward each ROI using guidance information supplied by the system’s monitor and precomputed navigation plan. During navigation, the physician uses the foot switch to invoke commands that update the guidance information. If desired, the physician can also invoke a position-verification mechanism to confirm and, if indicated, correct the bronchoscope’s position.

The system builds upon methods developed for our previous technician-assisted VBN system, which has been shown to provide effective guidance in live patient procedures [15], [16] and which also serves as the basis for the LungPoint VBN system [12], [17].

The work flow above is similar to that of our previous system (cf., [16, Fig. 2]), but incorporates two major innovations to accommodate hands-free operation. As highlighted in Section I, the new innovation of procedure planning is navigation plan computation, delineated by Algorithm 1. The navigation plan consists of a series of bronchoscope maneuvers abiding by the rotate-flex-advance paradigm. The major innovation of image-guided bronchoscopy is a new guidance strategy, given by Algorithm 2, and a complementary graphical user interface (GUI) especially tailored for displaying salient guidance information. The guidance strategy translates the previously computed navigation plan into navigation instructions displayed by the system GUI. The guidance strategy also includes the new position-verification mechanism. All of these coordinated elements then enable the physician to easily perform autonomous guidance via the foot switch.

Fig. 2.

Fig. 2

Possible bronchoscope maneuvers at a branch bj for the case when the blue-line route r appears on the view plane’s left side. (a) Beginning position at bj. Two rotate-flex-advance sequences are possible at this starting location: (b) rotate CW, flex down, advance; or (c) rotate CCW, flex up, advance.

The rest of this section is organized as follows. Section II-A first introduces fundamental conceptual elements and mathematical constructs. Sections II-B and II-C elaborate more on the two-stage work flow summarized above. Finally, Section II-D concludes with a discussion of implementation issues.

A. Conceptual Elements

For this paper, we follow and enhance when necessary some of the notational conventions of Gibbs et al. [16]. To begin, let the patient’s high-resolution 3-D chest CT scan define the virtual space, while the patient’s actual chest anatomy defines the real space. When navigating through the airways of the CT-based virtual space, we use a virtual camera, or virtual bronchoscope, to image the virtual airway tree’s interior [13], [15]. Similarly, the “real” bronchoscope camera images the patient’s airway tree interior during the procedure. Let ICTΘ denote the virtual bronchoscope’s view rendered at pose Θ within virtual space, where the virtual camera’s pose Θ is given by the standard six-parameter vector defining the pose’s 3-D spatial position and Euler angles. View ICTΘ, which is a CT-derived endoluminal rendering of the airway-tree interior, will be referred to as a VB view. Analogously, let IVΘ represent the bronchoscope camera’s view at pose Θ within the patient’s airway tree of real space. Each 2-D video frame of the bronchoscope’s live video stream gives such a view.

As is done in all IGI bronchoscopy systems, the airway centerlines serve as the natural pathways for navigating the airway tree [12], [16], [21], [35], [36]. We represent the centerlines by a discrete set of airway branches bj, where bj consists of an ordered discrete set of ωj view sites; i.e.,

bj={vj,1,vj,2,,vj,ωj} (1)

where vj, i is the ith view site constituting bj. For our work, a view site is specified by its 3-D location within virtual space, along with vectors defining the field of view and parameters measuring the local branch angle and branch diameter [16]. Finally, each branch bj has an associated branch length lj, defined as the Euclidean distance between the first and last view sites of bj.

For each ROI, let r denote a route that defines a pathway through the airway tree leading to the ROI. Route r consists of an ordered set of branches, decomposed into view sites per (1), leading from the trachea to a final destination near the ROI, i.e.,

r={b1,b2,,bj,,ba,ba+1} (2)
={v1,,v1,ω1,v2,2,,v2,ω2,v3,2,,v3,ω3,,va,ωa,va+1,2,,vD} (3)

where vD denotes the destination view site. All routes begin with the airway tree’s unique tracheal branch b1. Regarding (2), route r by convention consists of a complete branches and a partial (a + 1)th branch. Per (1), the final view site of branch bj, namely vj, ωj, also equals the first view site vj+1,1 of bj+1. Note, of course, that the value a can differ for different routes. Equation (3) will be useful, because it highlights route r’s a endpoints

vj,ωjvj+1,1,j=1,2,,a. (4)

As discussed in Section II-B, the method for navigation plan computation (Algorithm 1) uses view sites near the locations (4) to observe successive branch bifurcations along the route.

Given the definitions above, the goal of image-guided bronchoscopy is to assist the physician in navigating the “real” bronchoscope through the patient’s airway tree so that he/she follows the preplanned airway route r defined in virtual space. This suggests that, as the virtual bronchoscope moves along route r, the real bronchoscope should appear to be at the same location. In other words, during the live procedure, the bronchoscope’s video stream should appear to stay approximately synchronized, or registered, with the virtual bronchoscope’s VB view sequence along r.

As Section II-C will make clear, our system’s guidance strategy (Algorithm 2) expects this synchronization to occur near r’s bifurcation points (4), per the relation

IVΘjICTΘj,j=1,2,,a (5)

where IVΘj is a frame in the video stream, ICTΘj is a VB view, and

  • Θj ≜ virtual-space pose near branch bj’s endpoint vj, ωj on route r

  • Θj ≜ real-space pose approximately corresponding to the same viewpoint as Θj.

The physician, at times, may also use the system’s position-verification mechanism to further validate bronchoscope location.

B. Procedure Planning

Procedure planning begins with 3-D CT Analysis and concludes with navigation plan computation, as given by Algorithm 1.

Three-dimensional CT analysis employs previously validated methods described in [15], [16], [36]–[42]. To begin, the physician selects target ROIs on the patient’s chest CT scan. Next, automated operations derive a 3-D airway-tree model, consisting of the segmented airway tree, airway endoluminal surfaces, and the airway tree’s branch structures, plus other virtual-space quantities required by image registration. Note that the airway endoluminal surfaces delineate the navigable airways observable in virtual space. Further processing then uses known bronchoscope device constraints and the 3-D airway-tree model to derive an optimal route for each ROI. As discussed in detail in [16], this results in a route r that is optimal in two respects.

  1. It delineates an airway pathway that can accommodate the specified bronchoscope along its entire length, i.e., bronchoscope navigation is feasible along the entire route.

  2. It terminates closer to the ROI than any other potential airway pathway.

In past work, we used these routes in tandem with our previous VBN system to perform successful bronchoscopic guidance for lung-cancer patients [15], [16]. Using the system, the physician previewed a “VB” by playing a cine movie of the VB view sequence suggested by the CT-derived route [16], [44], [45]. While this preview nominally illustrates the proper bronchoscope trajectory to execute during later live navigation, two issues remain. First, the movements suggested by the movie preview do not necessarily translate into easily executed physical bronchoscope maneuvers. This leads to the second issue: the system—like all other existing VBN systems—requires an attending technician to operate during the live procedure.

Our new system alleviates these issues by precomputing a navigation plan consisting of natural bronchoscope maneuvers. The navigation plan in turn smoothly integrates into a hands-free guidance strategy. The remainder of this section describes navigation plan computation, while Section II-C discusses the guidance strategy.

Two notions motivate the construction of the navigation plan: 1) the inherent rigidity of the chest anatomy; and 2) the device mechanics enabling bronchoscope maneuverability.

Regarding the anatomy, we point out that the airway tree is torsionally rigid, a feature used by Bricault et al. to establish an initial estimate of bronchoscope position [30]. This feature implies that as long as the real bronchoscope moves along branch bj+1 from bifurcation vj+1,1 = vj,ωj to its successor bifurcation vj+1, ωj + 1 with a consistent initial roll angle and intrabranch rotation, then the relative orientation between the airway bifurcations observed in video views IVΘj and IVΘj+1 —and by analogy the virtual bronchoscope’s corresponding VB views ICTΘj and ICTΘj+1 —does not change, regardless of chest/breathing motion or patient cough.

Regarding device mechanics, Fig. 1 illustrates the basic construction of a bronchoscope. The physician maneuvers the bronchoscope by holding the device handle in his/her hand. Before beginning airway navigation, the physician first places the insertion tube, the main body of the bronchoscope, into the patient’s nose or mouth and advances it to the trachea. The physician now has three means for maneuvering the device through the airway tree:

Fig. 1.

Fig. 1

Bronchoscope construction and its usage for rotate-flex-advance maneuvers. (a) Parts of a bronchoscope. (b) Thumb positions on the bronchoscope control lever for flexing the bronchoscope tip up or down [top] and corresponding shapes of the bronchoscope tip [bottom].

  1. Rotation, by twisting the wrist clockwise (CW) or counterclockwise (CCW).

  2. Articulation, by flexing the thumb on the control lever to move the device’s tip up or down.

  3. Translation, by either advancing (pushing forward) or withdrawing (pulling back) the device.

Note that a bronchoscope cannot be moved laterally within the airways. Hence, as illustrated in Fig. 2, bronchoscope movements must follow the rotate-flex-advance procedure [33], [34]. For example, when the bronchoscope is positioned at a bifurcation, the physician must first rotate the bronchoscope so that the desired lumen region for the next branch appears at either the top or bottom of the video frame. Subsequently, to navigate the bronchoscope into the desired branch, the physician must next flex the bronchoscope’s articulating tip either up or down and then advance the bronchoscope into the desired airway. Fig. 2 illustrates the two rotate-flex-advance possibilities for the case when the route appears in the left half of the view plane at branch bj. Notice the differing route locations for branch bj+1 after applying these maneuvers, as shown in Fig. 2(b) and (c). In Fig. 2(b) and (c), the first view represents a view in branch bj after rotation, while the second view depicts the resulting view after performing the flex and advance into branch bj+1. Carefully note that these figures are correct: when the bronchoscope is rotated CW (CCW), the observed view appears to rotate CCW (CW).

As a further limitation, the bronchoscope cannot be rotated in the same direction indefinitely along a route, because the physician’s wrist cannot accommodate such a twist! The physician can only twist the hand that holds the bronchoscope over a limited range. Furthermore, three other bronchoscope connections limit the allowable twist: 1) the bronchoscope’s electrical connection to the video processor; 2) an ancillary device channel necessary for pumping/extracting saline and anesthesia through the bronchoscope insertion tube; and 3) the bronchoscope’s working channel, used for inserting biopsy needles and other instruments.

Given these limitations, the physician constrains his/her wrist position α to a feasible ±90° range about the neutral wrist position. The neutral wrist position denotes the α = 0° position, signifying that the hand is aligned with the forearm. This gives rise to three notable extrema conditions (see Fig. 3).

Fig. 3.

Fig. 3

Notable wrist positions when holding a bronchoscope. (a) Neutral position (α = 0°). (b) Max. CW twist (α = +90°). (c) Max. CCW twist (α = −90°).

  1. The wrist is in neutral position (α = 0°). Subsequent rotations can be either CW (limit = +90°) or CCW(limit = −90°).

  2. The wrist has been twisted a maximum amount in the CW direction, i.e., α = +90°. Any subsequent rotation must be CCW and can be in the range [−180°, 0°].

  3. The wrist has been twisted a maximum amount in the CCW direction, i.e., α = −90°. Any subsequent rotation must be CW and can be in the range [0°, 180°].

In this discussion, we use the convention that a CW rotation is represented by an angle >0°, while a CCW rotation takes a value <0°. Note that as the bronchoscope is navigated through each branch of a route, multiple rotations will be performed. The angle α, taken relative to a clock, keeps track of the bronchoscope’s current cumulative rotation angle—or, equivalently, the wrist’s position—along the route, where −90° ≤ α ≤ +90° at all times along the route in line with the wrist’s rotation limits.

Finally, note that as one advances along an airway route, the successive airway branches tend to become shorter, i.e., lj+1 < lj. For example, from extensive human studies, we have observed the following mean branch lengths (trachea = generation 1) [16], [46]: generation-2 right-main bronchus, 26.1 mm; generation 3, 17.4 mm; generation 4, 11.6 mm. Or, stated differently, on the order of 30% or more of encountered airway branches bj can have a length lj < 10 mm. Note that many such short branches form components of successive close bifurcations which essentially appear as trifurcations. As a practical matter, excessively short branches do not warrant a specific rotate-flex-advance maneuver. In fact, maneuvers for short branches can actually be distracting or impossible to do, such as those near what appears to be a trifurcation. For such situations, the physician essentially skips short airways and navigates through two or more consecutive airways in one rotate-flex-advance sequence. Therefore, by taking into account the limitations imposed by the anatomy and the bronchoscope, it is possible to precompute a navigation plan consisting of a sequence of bronchoscope maneuvers to perform along route r.

Algorithm 1 presents our approach for navigation-plan computation. The algorithm’s inputs are the CT-based 3-D airway-tree model, route r per (2) and associated branch lengths, plus three parameters constraining when to skip a short branch. The algorithm cycles through each branch bj, taking care to maintain the wrist position angle α within the feasible [−90°, 90°] range. The final output is a data structure

{Tj,Mj},j=1,2,,a (6)

representing the navigation plan for route r, where one set of maneuvers {Tj, Mj} exists for each branch bj. In (6), Tj and Mj represent a bronchoscope rotation (twist) and flex, respectively. Tj < 0° indicates a CCW twist command, while Tj ≥ 0° indicates a CW command, with the sign of Tj implicitly indicating the rotation direction. The value of Mj indicates the specified up or down flex maneuver. Note that the need to advance the bronchoscope to the next branch after each rotate-flex maneuver is implicit, since route r by definition consists of an ordered set of branches progressing deeper into the airway tree. Finally, a value Mj = NULL designates a skipped branch bj.

If branch bj is too short per minimum-branch length parameter lmin, then it is a candidate for skipping. Parameter skipmax limits the number of consecutive branches that can be skipped, while jmax disallows branch skipping for a route’s most peripheral branches (i.e., bj, j > jmax), as such branches help perform final bronchoscope localization near the ROI [8], [16].

Algorithm 1.

Navigation Plan Computation.

1: Inputs:
 CT-based 3D airway-tree model
 route r = {b1, b2, …}, branch lengths {l1, l2, …}
 preset parameters lmin, skipmax, jmax
2: α = 0°, skip = 0
3: for all bj, j = 1, 2, …, a do
4:   if ljlmin and
  skipskipmax and jjmax then
5:    Tj = 0°, Mj = NULL, skip = skip + 1
   // Skip bifurcation
6:    continue
7:   else // Derive rotate-flex maneuver for branch bj
8:    skip = 0
9:     ICTΘj=VB-view-compute(vj,ωj)
10:     θ=brach-rotation(ICTΘj,bj+1)
11:    if −90° ≤ θ ≤ 90° then // next branch lies in right-half of view plane
12:     θ1 = θ + 90°, θ2 = 90° − θ,
    right = TRUE
13:    else // next branch lies in left half of view plane
14:     θ1 = θ − 90°, θ2 = 270° − θ,
    right = FALSE
15:    if αθ1 > −90° then // CCW rotation
16:     Tj = −θ1, α = αθ1
17:     if right = = TRUE then
18:      Mj = DOWN // flex tip down
19:     else
20:      Mj = UP // flex tip up
21:    else // CW rotation
22:     Tj = θ2, α = α + θ2
23:     if right = = TRUE then
24:      Mj = UP
25:     else
26:      Mj = DOWN
27: return branch maneuvers {Tj,Mj }, j = 1, 2, …, a

Lines 7–26 of Algorithm 1 derive the rotate-flex-advance maneuvers for branch bj. Function VB-view-compute(·) computes a VB view ICTΘj observing branch-terminating site vj,ωj (the view is actually computed at a location 80% along branch bj’s length), while branch-rotation(·, ·) computes an angle θ ∈ [−90°, 270°]. Angle θ establishes the local angular relationship between bj and successor/child branch bj+1, i.e., whether bj+1 appears in the right-half or left-half of the u-v view plane of ICTΘj. θ is computed by first forming a line between child branch bj+1’s lumen-region center and the opposing child branch’s lumen center. Next, this line is projected to ICTΘj’s u-v view plane and centered about the origin. Angle θ equals the angle this line forms with ICTΘj’s positive u-axis. (Note that θ could also be derived by a cross-product between appropriate tangent vectors from view sites constituting bj and bj+1 [16], [30].) Given θ, angles θ1 and θ2 are computed, where −180° ≤ −θ1 < 0° and 0° ≤ θ2 ≤ 180°. Finally, θ1, θ2, and the current wrist position α are combined to derive the maneuvers {Tj, Mj}, where −θ1 signifies a CCW twist, θ2 signifies a CW twist, and Mj denotes the flex direction. The assigned Tj satisfies the wrist-action constraint, such that the downstream bifurcation leading to bj+1 appears north–south after rotating the bronchoscope an amount Tj. In this way, the physician can then flex the bronchoscope straight up or down per Mj and then advance into the next branch.

The online supplement gives a quantitative example of deriving a rotate-flex-advance maneuver sequence for the situation of Fig. 2. Finally, Fig. 4(a), described in the next section, gives a report-view example of a typical navigation plan for a four-branch route.

Fig. 4.

Fig. 4

Demonstration of the first complete rotate/flex/advance maneuver per Algorithm 2 (case B—see Table I). (a) System GUI display in preparation for the first rotation. (b) Display after completing the rotation. Endoluminal viewer labeled “1.)” indicates the state of this tool after invoking the “rotate” foot-switch command, while the panel labeled “2.)” shows the tool’s state after the physician rotates the bronchoscope as instructed. The views of the other two tools do not change during this maneuver. (c) System GUI state after invoking the “advance” foot-switch command—instructions appear for performing the required flex-advance command.

C. Image-Guided Bronchoscopy

Upon completing procedure planning, the VBN system can be wheeled into the bronchoscopy suite. After patient prepping, image-guided bronchoscopy can then proceed. This involves 1) system initialization followed by 2) live bronchoscopy.

To perform system initialization, we first interface the system to the bronchoscope’s output video source and invoke the system software, including the software driving the foot switch. Next, the patient’s CT scan, previously computed CT-based 3-D chest model, and navigation plan are loaded. The foot switch, positioned on the floor during bronchoscopy, has its three switches programmed with the following functions: 1) “rotate/advance,” to invoke rotation and advance commands; 2) “verify,” to invoke location verification; and 3) “back,” to enable backing up to a previous airway branch. Note that the foot switch is the preferred method for system operation. However, we also provide analogous keyboard and mouse commands that can override the foot switch. In addition, the bronchoscope’s video monitor is always available. These redundancies provide important fail safes in case of system failures, software crashes, or unexpected adverse events. We also emphasize that during our pilot human studies (see Section III-B), a technician was always available for assistance.

The final system initialization operation involves navigation plan preview. As we had done in our earlier technician-guided efforts, this preview gives the physician an opportunity to effectively rehearse the planned bronchoscopic routes virtually before performing the live navigation [16], [44], [45]. The physician can do this virtual rehearsal on either a portable tablet or the guidance computer.

The remainder of this section focuses on the elements constituting live bronchoscopy. We first describe the basic form of the system’s GUI. We then discuss Algorithm 2, the computational algorithm underlying the guidance strategy, and conclude with a discussion of the live guidance process.

For each route r, the system monitor’s GUI is initialized to present three continuously synchronized visualization tools, as shown later in Fig. 4(a).

The Navigation Plan Tool depicts the complete navigation plan for all pertinent airways. Each row gives a pair of VB views for each branch bj, based on the precomputed navigation maneuvers {Tj, Mj}. The “Arrive” column indicates the arrival/starting position for branch bj after completing the prescribed rotate-flex-advance sequence for branch bj−1. The transition between the “Arrive” and “Advance” columns indicates the necessary rotation Tj for bj, with the “Advance” column depicting the result. The “Advance” column view then pictorially implies the necessary bronchoscope flex Mj maneuver, with the result of the implicit advance maneuver from branch bj to branch bj+1 being depicted in the “Arrive” view of the succeeding row.

Thus, a rotate-flex-advance sequence for bj is represented as follows. During guidance, a green box highlights the VB view corresponding to the current location of interest. The “Arrive” column VB view corresponds to the beginning guidance position near bifurcation vj, ωj. CW and CCW instructions in between a view pair indicates the necessary bronchoscope rotation. The “Advance” column VB view visually depicts the result of the preplanned rotation. Because the view clearly shows the desired position after the rotation, we do not explicitly give the rotation angle Tj. Next, by way of the blue line corresponding to r, the “Advance” column’s VB view visually indicates whether to flex the bronchoscope up or down when advancing the bronchoscope to branch bj+1.

The Endoluminal Viewer gives a display consisting of three sections.

  1. A top legend displaying either bronchoscope navigation instructions or the remaining navigation distance to the destination ROI.

  2. A video/VB-view pair consisting of the live bronchoscope video stream (left) and the virtual bronchoscope’s current VB view (right). The VB view also shows the preplanned blue route r, along with ancillary airway pathways in red.

  3. A second “frozen” video/VB-view pair that depicts either a “base camp” view for the most recent real- and virtual-space synchronization or an indication that the bronchoscope is in an erroneous location. Fig. 4(b) and (c) gives examples of this view pair. This view pair is present at all times during navigation except during the initial position for b1 [e.g., Fig. 4(a)].

Finally, the 3-D Airway-Tree Viewer depicts the CT-derived 3-D airway tree, preplanned guidance route r in blue, the other ancillary airway centerlines in red, the virtual bronchoscope’s current 3-D position within the airway tree (tan cylinder and green needle), and the ROI.

During bronchoscopy, Algorithm 2 constitutes the guidance strategy driving system operation. Drawing upon the previously computed CT-based tree model, route r, navigation plan (6), and the live bronchoscope video stream, the algorithm begins with the system situated in the trachea b1 of virtual space. The remainder of the algorithm then runs through guidance instructions for the remaining route branches. For each branch bj, the physician runs through a complete rotate-flex-advance maneuver set or, if desired, the optional bronchoscope position-verification mechanism.

Regular typeface lines denote commands run by the guidance computer, e.g., line 2. Italicized lines in Algorithm 2 indicate two types of physician interactions. First, instructions such as “Move bronchoscope and “Rotate bronchoscope indicate expected, but optional, bronchoscope movements. Second, bold italicized instructions of the form Invoke<···> command” involve mandatory foot-switch commands (e.g., line 2)—the system’s state, as depicted by the GUI, will not change until the physician invokes the required command. The algorithm uses four distinct commands controlled by the foot switch: “rotate,” “advance,” “verify,” and “back.” Note, however, that the “rotate” and “advance” commands always must be used in succession. Thus, one foot-switch position handles both commands, and a three-position foot switch is sufficient. Finally, three functions perform special operations.

  1. VB-rotate-movie (ICTΘj,Tj)ICTΘjT plays a VB-view movie in the Endoluminal Viewer showing current “Arrive” VB view ICTΘj rotating an amount Tj and stopping at “Advance” view ICTΘjT. This function also causes the system state (i.e., the virtual bronchoscope) to move to the row j “Advance” view of the navigation plan.

  2. VB-advance-movie (ICTΘjT,bj+1)ICTΘj+1 plays a VB-view movie in the Endoluminal Viewer starting at current “Advance” view ICTΘjT and ending at “Arrive” view ICTΘj+1 of the next route branch. In addition, the function causes the system state to move to the row (j+1) “Arrive” view of the navigation plan.

  3. global-register (IV,bj)ICTΘ performs a global registration between the bronchoscope’s current real-space position as depicted in video frame IV relative to the VBN system’s current state in virtual space in branch bj. The output ICTΘ depicts the optimal VB view corresponding to IV along with a virtual space branch index k corresponding to ICTΘ.

Figs. 46 illustrate the guidance strategy, as dictated by Algorithm 2. The online supplement provides enlarged versions of these figures. The guidance strategy begins in the trachea (branch b1), with the following views appearing in the initial GUI display [see Fig. 4(a)].

Fig. 6.

Fig. 6

Error detection example for route r2, human case F. (a) The physician has clearly maneuvered into an incorrect airway, so she invokes “verify.” (b) System view after second invocation of “verify” command.

  1. The navigation plan tool highlights the virtual bronchoscope’s first VB view ICTΘ1 in the Arrive column for row 1. This view, which looks toward the main carina v1, ω1, also shows the route r in blue, other (erroneous) airway pathways in red, and the remaining distance 137 mm to the final destination in green.

  2. The endoluminal viewer also shows the destination distance in its top legend. In addition, it depicts the live video stream (left) and the VB view ICTΘ1 (right). Note in Fig. 4(a) that the video view indicates that the physician has already manually positioned the bronchoscope near the main carina, as suggested by the guidance system.

  3. The 3-D airway-tree viewer shows route r (blue) and the virtual bronchoscope’s current route position (tan cylinder and green needle) in b1 looking toward the main carina in the rendered 3-D airway tree. Other undesired airway pathways also appear in red.

Algorithm 2.

Technician-Free Image-Guided Bronchoscopy Strategy.

1: Inputs:
 CT-based 3D airway-tree model
 route r = {b1, b2, …}
 navigation plan {Tj, Mj}, j = 1, 2, …, a
 bronchoscope video stream
2: Present VB View ICTΘ1 // Initialization at main carina
3: Move bronchoscope so
IVΘ1ICTΘ1
4: for all bj, j = 1, …, a such that Mj ≠ NULL do
5:   // Perform rotate-flex-advance maneuver set
6:   Invoke “rotate” command
7:   Present VB-rotate-movie (ICTΘj,Tj)ICTΘjT
8:   Rotate bronchoscope so IV(ΘjT)ICTΘjT
9:   Invoke “advance” command
10:   Save base-camp pair {IV(ΘjT),ICTΘjT}
11:   Present VB-advance-movie (ICTΘjT,bj+1)ICTΘj+1
12:   Flex and advance bronchoscope so IVΘj+1ICTΘj+1
13:   // Position-verification mechanism
14:   if “verify” command invoked then
15:    Present targeting circles on video
16:    Move bronchoscope so a bifurcation fits inside circlesIV
17:    Invoke “verify” command
18:    global-register (IV,bj)ICTΘ
19:    Present base-camp pair {IV, ICTΘ}
20:    if depicted registration deemed correct
   then
21:     if bronchoscope in correct position
    then
22:      continue // Navigation on track
23:     else // bronchoscope deemed to be in incorrect position
24:      Pull back bronchoscope to branch bj
25:      go to 6 // Back up to last correct branch
26:    else // Depicted registration is incorrect
27:     go to 14
28: else if “back” command invoked then
29:   repeat
30:    j−1 → j
31:   until Mj ≠ NULL // Back up to first proper branch
32:   Present VB View ICTΘj // Present “Arrive” VB view for previous proper branch
33:   Pull back bronchoscope to indicated branch
34:   go to 6

The physician then invokes the “rotate” foot-switch command. This action causes the guidance display to highlight the CCW-rotated VB view at the main carina ICTΘ1T in the navigation plan tool’s advance column for row 1 [see Fig. 4(b)]. It also results in the endoluminal viewer’s top legend giving the CCW rotation instruction and current distance from the ROI. In addition, the Endoluminal Viewer plays a VB-view movie of the rotation from ICTΘ1 to ICTΘ1T. The 3-D airway-tree viewer’s display does not change, because the virtual bronchoscope does not advance during this action.

The physician now mimics the suggested CCW rotation, as shown in the second endoluminal viewer display in Fig. 4(b). The other GUI views, which follow the virtual bronchoscope’s movements, do not change. The first approximate real-space/virtual-space synchronization per (5) has now been completed between the guiding virtual bronchoscope and the physician’s “real” bronchoscope.

The physician next performs the necessary steps to complete the planned flex-advance maneuvers associated with branch b1. To do this, the physician first invokes the “advance” foot-switch command, resulting in the following actions by the three tools [see Fig. 4(c)]. The navigation plan tool highlights the arrive column view ICTΘ2 and destination distance (63.6 mm) in row 2 for b2. The endoluminal viewer completes the following actions:

  1. The top legend gives the “Move Up” instruction and destination distance 64 mm.

  2. The first video/VB-view pair plays a VB-view advance movie starting with ICTΘ1T and concluding with ICTΘ2 (right). It also continues displaying the current live bronchoscope video stream view (left). As shown, the physician has held the bronchoscope steady from the previous rotation.

  3. It creates a second frozen “base-camp” view pair, depicting the previously synchronized video/VB-view pair after the first rotation [i.e., bottom of Fig. 4(b)]. This pair is enclosed in a green box and labeled “Base Camp—Gen 1” to point out the last recorded registered position from the generation-1 branch b1.

The 3-D airway-tree viewer plays a movie showing the movement of the virtual bronchoscope from its old pose at Θ1T to its new pose Θ2 situated 64 mm from the destination. The physician now flexes and advances the bronchoscope as instructed, to complete the planned rotate-flex-advance maneuvers associated with b1. Subsequent maneuvers for branches b2, b3, …, follow a similar sequence. The online supplement gives a video demonstrating the complete guidance process for Fig. 4.

Ideally, if the physician properly mimics the suggested maneuvers, then a believable synchronization will appear to exist between the real bronchoscope and the VBN system’s virtual bronchoscope. In the course of many procedures, however, adverse events arise. These events can cause the bronchoscope’s video display to become temporarily obscured and/or for the bronchoscope to move slightly. As a result, the physician may now become unsure of the bronchoscope’s actual position. To help with such situations, the physician can draw upon the position-verification mechanism. Figs. 5 and 6, examples drawn from our human studies of Section III-B, illustrate the use of this mechanism.

Fig. 5.

Fig. 5

Use of the position-verification mechanism for route r8, human case D (see Section III-B). (a) System view after physician invokes the “verify” command. (b) System view after second invocation of “verify” command.

For the situation of Fig. 5, the physician appears to have positioned the bronchoscope incorrectly. Thus, she invokes the “verify” foot-switch command. This triggers a series of actions, beginning with two targeting circles being superimposed on the endoluminal renderer’s live video feed [see Fig. 5(a)]. These circles are derived from the known diameters of airway branches within a subtree ±3 airways about the most recently navigated branch bj, where the smaller circle encompasses the smallest bifurcation in the subtree and the larger circle accommodates the largest bifurcation. The physician next adjusts the bronchoscope so that a nearby bifurcation fits within the limits of the two circles and invokes a second “verify” command. This causes an automatic global registration between the displayed video frame IV and CT-based virtual space, focused on a subtree around the current branch bj. The endoluminal viewer’s revised base-camp pair now shows that the bronchoscope’s position was in fact correct and asserts this conclusion with the message “On Path. Move forward” [see Fig. 5(b)]. Notice that base-camp pair also fuses the blue guidance route r onto the frozen video view to further assert this conclusion [16]. In this way, the physician successfully verified her position and navigated forward confidently.

Note that the global registration function searches the 3-D airway-tree model’s centerline structure within the subtree to identify the airway branch b giving the closest match to IV— this denotes the derived global bronchoscope position illustrated in the base-camp examples of Figs. 5(b) and 6(b). Briefly, this search process occurs as follows.

  1. Derive the minimum-enclosing rectangle containing the airway-lumen regions depicted in IV.

  2. For each branch in the subtree about bj,

    1. Identify the view site along branch that gives VB view ICTΘv^ whose minimum-enclosing airway-lumen rectangle best matches the corresponding rectangle found for IV.

    2. Apply the inverse-compositional CT-video registration method of Merritt et al. between ICTΘv^ and IV to arrive at VB view ICTΘv^o with optimized virtual-space viewpoint Θv^o [15].

  3. Find the VB view ICTΘv^o over all subtree branches that best matches IV, using a least-squares error metric. The branch minimizing this metric gives the desired b and VB view ICTΘ.

As the global registration method outlined above is a major topic in its own right, please refer to the references for complete detail [42], [43].

If the identified branch b is not part of route r—i.e., the bronchoscope has gone erroneously off path, then global registration traces back through the subtree starting from b until it reaches a branch bkr. Next, an updated base-camp pair {IV, ICTΘ} is presented, where ICTΘ denotes the derived global bronchoscope position b in virtual space. Depending on the derived b and bk, global registration also returns a requisite directive indicating whether the physician is on the correct route, as in Fig. 5(b), or whether the physician must perform a specific pull-back operation to correct an erroneous bronchoscope position.

Fig. 6 illustrates a situation, where the physician did in fact maneuver the bronchoscope into an incorrect branch. Drawing upon the system’s position-verification mechanism, the physician invokes “verify,” causing the targeting circles to appear on the video frame [see Fig. 6(a)]. After adjusting the bronchoscope as needed so that the displayed bifurcation accommodates the limits set by the concentric circles, a second invocation of “verify” produces the global-registration result of Fig. 6(b). The revised base-camp pair clearly shows a correct real- and virtual-space registration at a location that is off the desired blue route—significantly, the desired blue route does not appear in either base-camp view. In addition, the message “Off Path! Fall back 1 generation!” appears. To correct this navigation error, the physician follows the given instruction by pulling back the bronchoscope 1 airway generation, thereby returning to the closest correct branch on the route. Later offline inspection of the procedure video revealed that the physician had failed to perform a proper rotation as directed by the system for branch b2 and, hence, proceeded into an incorrect airway. After correcting this error per Fig. 6(b), the physician correctly reached the final route destination (Fig. 12 in the online supplement).

Because the airway tree consists of a hierarchical connected series of branches, it is important to note that all navigation errors can be corrected by pulling back the bronchoscope to a correct lower-generation airway. This observation motivates the position-verification mechanism’s back up function. Invoked by the “back” foot-switch command, this function is employed for correcting navigation errors, as in Fig. 6. After invocation of the “back” command, the system’s virtual bronchoscope and GUI display back up to the previous airway. Next, the physician pulls back the real bronchoscope to the indicated airway, presumably getting the bronchoscope back onto the proper route. Navigation then proceeds as before. As another point, global registration can be sensitive to IV’s quality, a situation that could occur for especially problematic adverse events. Thus, the back-up function also is useful as a simple safeguard to overcome such situations.

In general, the physician easily performs all commands with the foot switch. We strongly caution, however, that if the procedure is in the midst of an adverse event, the physician is advised to let the event pass before moving the bronchoscope or invoking any command. When the physician approaches the end of the route—i.e., when the bronchoscope is within a few airway branches of destination vD —the system begins to display additional ROI localization information [16]. These features, developed in previous work, are illustrated by Fig. 12 in the online supplement.

D. Implementation Comments

All procedure planning computations occur offline prior to the live procedure. The 3-D CT analysis operations typically take on the order of 5–10 min of computation on the guidance computer, while navigation plan computation per Algorithm 1 requires under a few seconds. This assumes a typical high-resolution patient 3-D CT scan containing several hundred 512 × 512 axial-plane sections (typical voxel resolution ≈ (0.75 mm)3 ). Such CT resolution is typical for modern IGI systems for bronchoscopy and is essential for deriving airway routes having ≥ 4 airway generations [12], [16]. For our work, the spacing of view sites along a route is generally also of the order of 0.75 mm, and a typical human procedure involves ≤ 5 ROIs.

We defaulted Algorithm 1’s three parameters for skipping short branches to the following conservative values in our studies: lmin = 7.5 mm, skipmax = 1, jmax = 8. With these values, we never skip more than one branch in the navigation plan, even if two consecutive branches are shorter than 7.5 mm, a very modest restriction given typical airway lengths. Also, the large majority of the routes encountered in our human studies have a route depth shorter than eight airways; hence, the jmax restriction seldom arose (see Section III). In addition, per the physician’s preference (see Section III-B), our final system includes a user preference that forces the first rotation while in the trachea b1 to place the next route branch b2 at the top of the VB view ICTΘ1T. Other physician preferences could also be incorporated (see Section IV). Finally, note that Kukuk had previously suggested an automatic method for computing a series of maneuvers for bronchoscope navigation [47]. The method, however, was theoretically tailored to a robot, which has no human-related movement constraints, and was never put into practice.

As stated earlier, our VBN system expands upon the software and methods devised for our earlier technician-assisted VBN system. The 3-D airway-tree viewer used by our current system’s GUI is directly borrowed from the work of Merritt et al. [15] and Gibbs et al. [16]. The endoluminal viewer, however, required major upgrades to the Merritt/Gibbs efforts to accommodate the technician-free guidance strategy. Finally, the navigation plan tool is new. While the system tested by Asano/Shinagawa also offers static views of preplanned airway bifurcations to navigate, these views do not necessarily abide by any form related to how the physician performs bronchoscopy [10], [12]. Our tool, on the other hand, incorporates actual preplanned bronchoscope maneuvers, with accompanying VB views, abiding by the physician’s bronchoscopy training and other procedural constraints. Our tool also enables dynamic movie-form visualizations of the actions during a procedure.

Regarding the guidance strategy, our actual implementation incorporates more flexibility in the position-verification mechanism than encapsulated in Algorithm 2. First, the “verify” command can be invoked at any time, as can the “back” command, i.e., they serve as system interrupt commands. In addition, multiple back-ups can be performed during navigation. We excluded these features from the presentation of Algorithm 2 for clarity. Overall, the position-verification calculations take on the order of 1–2 s.

III. Results

This section presents a series of studies that address the following questions regarding the proposed system.

  1. Is it feasible and safe for the physician to navigate a bronchoscope without technical assistance?

  2. Is navigation success comparable to current technician-assisted VBN systems?

  3. Does the system offer any benefit in terms of efficiency and/or performance over current VBN systems?

  4. Does a physician judge the system to be intuitive and convenient to use?

The first three questions address basic system functionality and performance, while the last question considers the system’s acceptability from a human-factors perspective. The studies presented herein are summarized as follows.

  1. Section III-A presents phantom studies that help gain an initial appreciation of system functionality and efficacy in a controlled environment. It also gives a direct comparison of the proposed hands-free VBN system to a previously validated technician-assisted VBN system.

  2. Section III-B presents a human study that considers the safety, feasibility, and efficacy of the system in the live clinical environment for lung-cancer patients. It also compares our system to other technician-assisted VBN systems tested in previous clinical studies.

  3. Section III-C gives an ergonomic study, whereby a physician judges the system’s suitability for image-guided bronchoscopy.

All phantom and human studies involved performing live bronchoscopy. For these studies, we measured performance using the following three metrics, as used previously by other VBN system investigators [9], [10], [16], [48]:

  1. Bronchoscope insertion depth, defined as the number of airway branches (a+1) constituting the endobronchial (airway) guidance route r leading to an ROI, per (2).

  2. Navigation success, defined as correctly guiding the bronchoscope to the final airway branch ba+1 of a route.

  3. Guidance time, defined as the time required to guide the physician from the main carina (end of b1 ) to the final airway branch.

A. Phantom Studies

We performed two studies involving live bronchoscopy on phantoms, where a phantom served as the “patient.” Such studies allowed assessment of our system in a low-stress, controlled, known environment, free of patient-breathing issues and adverse events. The first study focused on basic system functionality, while the second study involved a randomized study, whereby a physician applied two different VBN systems: our new system and a previously validated technician-assisted system [15], [16].

Both studies drew upon two phantoms employed in previous studies [8], [15]. The phantoms consist of plastic airway-tree models prototyped from the airway endoluminal surfaces obtained from two 3-D CT scans of human patients (see Table I). They give realistic accurate models of the 3-D airway-tree interior.

TABLE I.

CT Scan Specifications for the Human Airway-Tree Phantom Cases

Patient Case number Scanner No. of axial-plane sections in scan Voxel Resolution (Δx, Δy, Δz) in mm
A Siemens Sensation-16 CT 706 0.67, 0.67, 0.5
B Phillips Gemini True Flight PET/CT 373 0.71, 0.71, 0.8

Both scans consisted of 512 × 512 axial-plane sections.

For these two cases, we performed procedure planning per Section II-B. In particular, we performed 3-D CT analysis to derive the 3-D airway-tree model and a set of preliminary routes rk for a total of 11 ROIs distributed over the left and right lungs in the CT scans. Next, navigation plan computation was performed for the routes to give the associated navigation plans in the form (6).Table II gives details on the routes. The descriptor “Location” in Table II refers to the location of the ROI associated with a particular route rk, as done by investigators in other studies [9], [10], [16], [48]. It equivalently denotes the location of the terminal airway branch ba+1 for a given route.

TABLE II.

Preplanned Guidance Routes rk for the Two Phantom Cases

Case Number Route Location Insertion Depth Route Length (mm) Guidance Time (min:sec)
A r1 LLL 2 144 0:28
r2 RML 2 121 0:33
r3 RLL 3 143 0:31
r4 RLL 6 165 0:51*
r5 LUL 5 167 0:41*
mean ± SD: 0:37 ± 0:08

B r1 RLL 8 214 1:20
r2 LLL 6 233 0:55
r3 RML 5 199 0:44
r4 RLL 12 264 1:28
r5 LLL 9 274 1:17
r6 LUL 5 213 0:46
mean ± SD: 1:05 ± 0:16

“Location” denotes the lung lobe containing the final airway ba + 1 of a route rk per (2). RUL = right upper lobe, RML = right middle lobe, RLL = right lower lobe, LUL = left upper lobe, LLL = left lower lobe. “Insertion Depth” gives the number of airway branches in a route. “Guidance Time” indicates the time required by the bronchoscopist to navigate each route using the proposed VBN system. A guidance time marked with a “*” indicates that the bronchoscope was unable to reach the final airway of a route due to physical limitations. “mean ± SD” combines guidance times over all routes for a case.

As can be seen from Table II, the routes span a variety of lung locations and lengths. Case A has a relatively small airway tree, with a route length mean ± SD (standard deviation) = 148 ±17 mm (range: 121–167) and insertion depths (number of airways per route) ranging from 2 to 6. Case B, on the other hand, has a large airway tree, with route length mean ± SD = 233 ± 28 mm (range: 199–274) and insertion depth ranging from 5 to 12.

1) Study 1—System Feasibility and Functionality

An experienced engineer served as the bronchoscopist, who operated the VBN system and performed image-guided bronchoscopy on both phantoms for the 11 predefined routes. An Olympus BF P180 true-color bronchoscope (distal diameter = 4.9 mm) was used to perform bronchoscopy. During all test procedures, we recorded the bronchoscopic video and guidance system’s computer display; these data were used for later data analysis.

Hands-free navigation proceeded smoothly for all routes, with the engineer successfully navigating all routes for the two phantoms, i.e., navigation success = 100%. Table II lists the guidance times, where guidance time was measured as starting at the first invocation of the foot switch at branch b1 and ending when the final airway branch was reached. Because phantom case A corresponded to a patient with relatively narrow-diameter airways, the bronchoscope’s diameter exceeded the diameter of the final airways constituting routes r4 and r5. Hence, because of the phantom’s rigidity, we could only maneuver the bronchoscope to the penultimate airways of these routes. (In clinical situations, the physician can often force the bronchoscope through the flexible distal airways.) For case A, guidance time mean ± SD = 0:37 ± 0:08 (range: 0:28–0:51), while for the larger B phantom, the guidance time mean ± SD = 1:05 ± 0:16 (range: 0:44–1:20).

The results clearly indicate that navigation time increased as the number of airways constituting a route increases, regardless of a route’s length. This observation implies that much of the actual navigation time is taken up by executing the foot-switch commands and then performing the indicated maneuvers. Conversely, route length gave little indication of the expected guidance time.

Fig. 4 gives sample system display views for route r2 of case B. In Fig. 4(a), notice that the navigation plan tool implies that route r2 consisted of four maneuver sets {Tj,Mj }. Table II, however, indicates that the complete route consisted of six airway branches. As it turned out, two of r2’s constituent branches failed to meet the length restriction lmin > 7.5 mm. Hence, Mj = NULL for these branches resulting in their being skipped in the navigation plan.

Fig. 4 demonstrates that precise registration of the live video source and presented VB views is not required, in line with (5). The structural information conveyed by the synchronized view pairs makes it clear that the bronchoscopist properly abides by the guidance information. Also, the navigation plan tool’s display conveys telling update information to the bronchoscopist related to current bronchoscope position and orientation in addition to information for subsequent maneuvers.

The online supplement illustrates a use of position verification for case B. Section IV offers further insight.

2) Study 2—Randomized Comparison of Two VBN Systems

Overall, Study 1 offered encouragement that the system functions satisfactorily and that hands-free guided bronchoscopy with a VBN sytem is feasible. It does not, however, give any indication as to how it compares to a typical technician-assisted VBN system. Study 2 gives such a comparison.

For the study, a pulmonary physician applied two different VBN systems to the task of bronchoscopic navigation through the two phantoms: the proposed hands-free system and the technician-assisted system of Merritt and Gibbs [15], [16]. For the study, the physician performed two separate trials of applying each VBN system to both phantoms. To minimize possible learning effects, we randomized the order that a system was used. Within a given trial for a VBN system, we also randomized the order in which the phantoms underwent bronchoscopy and the order of routes navigated for a particular phantom. We did take care, however, to guarantee that consecutive routes did not terminate in the same lung.

The study was completed in five separate sessions spaced over a five-week period. Three sessions involved using one VBN system for both phantoms. Due to time constraints, two sessions involved using a VBN system on only one phantom. Trials with the small phantom were performed with an Olympus XP160F ultrathin bronchoscope (distal diameter=2.8 mm); this enabled unhindered navigation to the final airway branch for all routes. Trials with the large phantom drew upon an Olympus MP160F bronchoscope (distal diameter = 4.0 mm).

For each session, the physician received “refresher” training on how to use the guidance system before performing the indicated trial. For trials involving the hands-free VBN system, the physician performed guided bronchoscopy “solo.” For trials involving the technician-assisted VBN system, the physician worked with an experienced technician. Before a trial, the physician and technician established a communication rapport.

Table III gives the results. In the table, we reordered the routes by number of branches per route and route length to emphasize how guidance time appears to increase as the number of route branches increases. Because of the simplicity of phantom navigation (“cooperative” patient, no breathing, rigid structure), navigation success was 100% over all trials. For 18/22 trials over the two phantoms, the new system enabled faster navigation than the technician-assisted system. In fact, the hands-free system enabled a significant decrease in navigation time for phantom 1 by 14 s (38%) on average (p = 0.003, per a two-sample unequal-variance paired t test applied to the two datasets), while for phantom 2, this average decrease was 9 s (18%) on average (p = 0.019).

TABLE III.

Results for Randomized Phantom Study

Case Number Route Insertion Depth Hands-Free (min:sec) Tech-Assisted (min:sec)
trial 1 trial 2 trial 1 trial 2
A r2 2 0:12 0:12 0:28 0:22
r1 2 0:12 0:13 0:25 0:22
r3 3 0:17 0:25 0:38 0:23
r5 5 0:26 0:38 1:05 0:50
r4 6 0:41 0:35 0:49 0:44
mean ± SD: 0:23 ± 0:12 0:37 ± 0:15

B r3 5 0:31 0:31 0:48 0:35
r6 5 0:34 0:43 0:47 0:32
r2 6 0:34 0:42 0:46 0:37
r1 8 0:52 0:39 1:09 1:04
r5 9 1:01 1:11 1:13 1:14
r4 12 1:12 1:00 1:44 0:59
mean ± SD: 0:48 ± 0:15 0:57 ± 0:21

“Hands-Free” and “Tech-Assisted” columns give the guidance times for each guidance trial for the indicated VBN system. The values for mean ± SD give mean and standard deviation information over all measurements for a given system.

As an interesting side point, Tables II and III indicate that the physician always performed hands-free navigation faster than the trained technician for the phantoms. This is probably attributable to the physician’s far greater bronchoscopy experience and more extensive anatomical knowledge.

B. Human Studies

The phantom studies appear to suggest significant benefits for the hands-free system. A realistic test with live patients in the clinical environment, however, would provide a more realistic test. To this end, we performed a human study of the system.

The study goals were to ascertain system safety, functionality, and efficacy under the demands of the clinical environment. Under an IRB protocol approved by our University Hospital, we enrolled and consented nine patients scheduled for lung-cancer staging bronchoscopy. The patient group included seven women and two men (ages, 50–74 years). Either a Siemens Sensation 40 or Siemens Definition CT scanner was used for producing the 3-D CT scans. All scans consisted of a contiguous series of 2-D axial-plane sections (512 × 512 voxels per section), with the number of sections per scan ranging between 274 and 760. Axial-plane resolution (Δx, Δy) ranged between 0.62 and 0.74 mm. Section spacing Δz = 0.5 mm for all cases, except for case C where Δz = 0.8 mm.

Before beginning the human studies, we conducted a training session with the physician using the phantom cases. Training covered both the procedure planning and image-guided bronchoscopy phases of system operation, with an especial emphasis on hands-free guidance operation. In addition, during the time frame of the human studies, we frequently carried out refresher training sessions, as is necessary when becoming familiarized with a new technology. Also, such refreshers proved necessary, because two to four week gaps tended to exist between human studies.

Forty-three total ROIs were selected for the nine cases, distributed over both lungs in peripheral locations (insertion depth ≥ 3 for all ROIs). Thirty-eight ROIs were defined as part of the comprehensive airway exams conducted by the physician during a typical procedure; five ROIs facilitated bronchoalveolar lavage (BAL). Procedure planning assumed that either an Olympus BF P180 true-color bronchoscope (distal diameter=4.9 mm) [eight cases] or an Olympus BF-1T180 bronchoscope (distal diameter =6.0 mm) [one case] would be used for live bronchoscopy. As a result, prior to bronchoscopy, procedure planning revealed that two ROIs could not be accessed by the selected bronchoscope, a conclusion our optimal route planning methods are able to ascertain [16]. Hence, these ROIs were excluded from the live tests.

Eight cases took place in the bronchoscopy suite, dedicated to minimally invasive endoscopic procedures, and one case occurred in the high-demand operating room. All cases also entailed additional necessary clinical procedures beyond the scope of our study’s goals, such as transbronchial biopsy and/or endobronchial ultrasound. We did not use our system during these phases of a patient procedure. As a standard practice, the physician reviewed the preplanned routes for each case before the start of bronchoscopy [16], [44], [45]. In addition, we recorded all procedures, including the VBN system’s GUI display, to assist with data analysis afterward. For operating-room case J, two ROIs were situated beyond a large left-lung tumor to be resected. Thus, while the physician successfully navigated all but the last airways of these ROI routes during bronchoscopy, the medical team advised against completing these exams to avoid disturbing the tumor. Hence, these ROIs were also excluded from consideration.

Thus, the physician performed image-guided bronchoscopy for 39 ROIs, as summarized in Table IV. These ROIs were distributed over the five lobes of the two lungs as follows: 7 ROIs, RUL; 5, RML; 9, RLL; 10, LUL; 8, LLL. For the 39 associated routes, insertion depth mean ± SD = 4.9 ± 1.6 airways (range: 3–10), while route length mean ± SD = 171 ± 25 mm (range: 124–245).

TABLE IV.

Live Human Study Cases Used for Testing the Proposed Hands-Free VBN System

Case # No. of Routes Insertion Depth Range over all routes per case
Route Length (mm) Guidance Time (min:sec)
C 4 4–7 151–159 0:23–0:46
D 4 4–7 177–180 0:26–1:29
E 4 3–4 166–201 0:09–0:44
F 4 3–4 130–156 0:33–0:52
G 6 4–8 151–205 0:39–1:30
H 5 4–6 124–159 0:43–1:37
I 5 4–8 155–202 0:18–0:58
J 3 6–10 180–245 0:46–1:08
K 4 4–8 128–163 0:39–1:47

“No. of Routes” refers to the number of routes considered for a case.

Table V summarizes study results, while Figs. 56 and Figs. 9 and 12 in the online supplement give example views from the study. Per Table V, navigation success = 97% with a guidance-time mean ± SD = 0:52 ± 0:23. The one navigation failure arose for route r3 of the very first case. The system’s suggested downward maneuver for the first main-carina bifurcation was counterintuitive and proved awkward for the physician to perform, resulting in the error. We, therefore, after this case, added a user option which forces the first maneuver to always be directed upward, as mentioned in Section II-D. After this design enhancement, we observed no further navigation issues.

TABLE V.

Human Study Results for the Hands-Free VBN System and for the Technician-Assisted VBN Systems Tested by Asano/Shinagawa and Gibbs et al. [9], [10], [16], [49]

System No. of Routes Navigation Success Insertion Depth Guidance Time (min:sec)
mean ± SD range mean ± SD range
Hands-free 39 38 (97%) 4.9 ± 1.6 [3,10] 0:52 ± 0:23 [0:09, 1:47]
Asano [9] 38 36 (95%) 6** * * *
Shinagawa [49] 85 * 5.7 ± 1.3 [3,10] * *
Shinagawa [10] 71 * * * 5:18 ± 2:00 *
Gibbs [16] 69 63 (91%) 7.1 ± 2.4 [3,13] 5:15 ± 5:06 [0:25, 32:07]
Gibbs, large bronchoscope 16 15 (94%) 4.7 ± 1.4 [3,8] 3:53 ± 3:54 [0:41, 12:30]

The “Hands-Free” system results drew upon the protocol described in the text and patient population of Table IV. Results for the other systems were gleaned from their respective publications. “No. of Routes” denotes the total number of airway routes navigated in a study over all enrolled patients. “Navigation Success” gives both the number of routes successfully navigated and success percentage. “Insertion Depth” and “Guidance Time” columns report the overall mean ± SD and range of these quantities.

*

Item not explicitly reported by study.

**

Median insertion depth per [9].

Fig. 12 of the online supplement illustrates the extra guidance information presented by the system when navigation reaches the end of a route. As discussed more fully by Gibbs et al., when approaching the final destination, the ROI (green) begins to appear in the VB view. In addition, a large arrow also appears indicating the final airway position for performing a possible needle biopsy [16].Overall, the system proved safe and effective over all studies, with no adverse events unduly affecting system operation.

Table V also compares the proposed system’s results to human studies performed previously with the technician-assisted VBN systems of Asano/Shinagawa and Gibbs et al. [9], [10], [16], [48]. All studies considered a variety of peripheral ROIs (insertion depth ≥3 airways). Asano/Shinagawa applied the commercial Bf-NAVI VBN system by Cybernet Systems in their studies [12] and drew upon ultrathin bronchoscopes exclusively. The Gibbs et al.’s study considered either ultrathin or larger bronchoscopes, depending on the clinical need for an ROI.

The table indicates that our proposed system enabled navigation success comparable to the other technician-assisted systems. Table V also appears to indicate that our system enables substantially faster bronchoscopic navigation than the other systems. A more careful examination is needed, however, given the varying insertion-depth ranges considered by the studies. First, the Shinagawa’s study reported insertion-depth data, but did not report guidance time [48], while the Shinagawa’s study reported a guidance time, but did not report insertion depth [10]. They did, however, both report nearly identical VB-view reconstruction depths (the maximum preplanned depth they could insert the bronchoscope live): 7.5 ± 1.2 ([48]) versus 7.3 ± 1.5 ([10]). Given that both Shinagawa studies occurred in the same time frame at the same institution, it could be inferred that both studies likely produced comparable insertion depths and guidance times. As an important distinction, because small 2.8-mm-diameter ultrathin bronchoscopes enable deep navigation into the airway tree, the Asano/Shinagawa and Gibbs studies on average considered deeper airway routes than we could in our current study, which utilized a large 4.9-mm-diameter bronchoscope. Thus, a comparison to the complete datasets of these studies does not strictly speaking offer a fair comparison to our system.

Fortunately, we had access to the Gibbs et al. outcome data and were able to parse out the subset of routes navigated with large bronchoscopes, as given by the entry “Gibbs, large bronchoscope” in Table V [16]. This gives a technician-assisted group that is more directly comparable to our current study’s data, based on insertion depth: 4.7 ± 1.4 (Gibbs) versus 4.9 ± 1.6 (proposed hands-free system). Using a two-tailed t test to make an inter-study comparison of these two datasets reveals that our proposed system enables a significantly faster guidance time (p = 0.009). In fact, the navigation speed-up on average was 3 min faster per route. Thus, our new system appears to potentially offer substantially faster navigation than both technician-assisted systems.

Interestingly, the new system seems to offer a much greater gain than expected, given the more modest results of the phantom studies. Evidently, this gain arose, because the phantom is an “easy” test involving a rigid motion-free “patient,” while the live human situation introduces considerably more unknowns and concomitant stresses to the procedure.

C. Ergonomic Study

As noted in Section I, the need for a trained technician has limited the acceptance of current VBN systems. Fortunately, the studies of the Sections III-A and III-B solidly establish the safety and performance of our proposed VBN system, while also suggesting that our new system offers substantially faster navigation—all without the extra overhead of the attending technician. However, as is well recognized in the field of IGI systems, ergonomic factors—i.e., factors influencing how a human interacts with the system—significantly impact the design and acceptance of an IGI system, especially since the physician is the ultimate judge of a system [49]–[51].

In an effort to address this issue, we performed a benchmark ergonomic study to assess the acceptability of our system for image-guided bronchoscopy. The study entailed two complementary objectives.

  1. Determine the percentage of suggested bronchoscope maneuvers that correspond to maneuvers preferred by the physician.

  2. Determine the physician’s ability to predict the actual maneuvers used during live bronchoscopy.

The first objective assesses the degree to which the physician agrees with our system’s suggested navigation plans. The second objective measures how well the physician can predict the way a live procedure will actually proceed. Both objectives draw upon the physician’s training regimen and clinical experience. These objectives also attempt to decouple any bias a physician may have, arising from experience with past established technologies. Finally, they measure how similarly the physician and VBN system “think.” Presumably, the more similar they are, the higher the likelihood the physician will accept the system.

All study benchmarking was performed by the attending physician who performed the prior human study. The study considered the data from the first eight cases of the human study, per Section III-B and Table IV. We restricted our attention to these cases, since they all occurred at least two weeks prior to the ergonomic study. This reduced the possibility of the physician remembering the routes navigated during the live procedures. In line with the two study objectives, the physician performed two distinct tasks with the data for each navigated route: 1) review the suitability of the navigation plan suggested by the system; and 2) predict how bronchoscopic navigation actually proceeded during the live study.

The study considered 37 ROI routes for the eight cases. These routes were distributed as follows: 6 routes, RUL; 5, RML; 8, RLL; 10, LUL; 8, LLL. We included the two left-lung routes excluded for the operating-room case J, as sufficient recorded data existed from the planning and live-procedure stages for these routes. One hundred and seventy-eight total branches constituted the 37 considered routes. The VBN system produced navigation plans consisting of 145 discrete maneuver sets, i.e., 33 short branches were skipped overall by the 37 navigation plans.

When assessing the suitability of a suggested navigation plan, note that if the physician decided that a maneuver at branch bj was nonideal, then all other subsequent suggested maneuvers along a route would also be nonideal. Hence, a higher premium naturally exists for maneuvers occurring early in a route. We emphasize that suggested maneuvers were not “wrong” per se, but instead deemed inconvenient or nonideal by the physician to perform during a live procedure. All routes were already known to traverse the airway tree correctly to the final ROI location. A so-called ideal route, though, also has “ideal” bronchoscope maneuvers associated with it at every juncture along its extent. (Recall, again, that the system helped the physician correctly navigate 97% of the preplanned routes during the human study.) Table VI of the online supplement summarizes the results. From the study results, we note the following observations.

  1. The portion of acceptable maneuvers suggested by the VBN system was 118/145 (81%).

  2. The average navigation plan gave 0.73 nonideal maneuvers per route.

  3. 26/37 routes (70%) gave acceptable preplanned maneuvers for all branches along a route.

  4. The physician correctly predicted the complete route actually used during the live procedure for 30/37 routes (81%).

  5. Both the VBN system and physician were in complete agreement for 24/37 routes (65%).

  6. For three routes, neither the system or physician predicted a complete ideal route.

Note that route r3 of the first live study C accounted for 6/27 nonideal maneuvers with one noted on the very first maneuver— and, as noted in Section III-B, this was the only route resulting in a navigation failure (and subsequent system upgrade). For the 11 routes identified as having at least one nonideal maneuver, four routes had a nonideal maneuver that occurred only on the last route branch. Such maneuvers are inconsequential, as the ROI is well localized and clearly visible when the last branch is reached. Thus, if these routes are also included as acceptable, then the overall rate of acceptable routes rises to 30/37 (81%).

IV. Discussion

The low success rate of standard bronchoscopy has been well documented [7]. In addition, the seminal 1994 Minami et al.’s study was the first to note that bronchoscopy skill variations can translate into a 2-to-1 performance difference between physicians [5]. The controlled study of Merritt et al. focusing on bronchoscopic navigation to peripheral lesions corroborated these observations, finding a 43% mean bronchoscopy success rate and a notably wide 20–70% performance range over 12 physicians [8]. Transthoracic needle aspiration is an alternative to bronchoscopy for lung-cancer staging, but it has a high pneumothorax rate and is not applicable to lymph-node biopsy [3], [6], [7].

Fortunately, recently proposed IGI bronchoscopy systems have been shown to greatly increase bronchoscopy success, while also reducing the skill variations between physicians. Lending support to this assertion, the Merritt et al.’s study demonstrated that navigation success rate rose from 43% for standard bronchoscopy to 96% using an IGI system [8]. Related to this observation, it is vital to realize that without image-based guidance, previous research has shown that a physician’s ability to correctly plan a route of depth 4 is only on the order of 40%, while a physician’s skill in navigating a bronchoscope to lesions of airway depth 6–8 is only on the order of 30% [4], [8]. Thus, it can be argued that the physician simply cannot satisfactorily plan or perform bronchoscopy to peripheral sites via standard techniques and that image guidance is essential. Furthermore, the performance benefits of IGI systems are making it imperative that they become a part of standard bronchoscopy practice. This is especially true, given the recent international effort to introduce lung-cancer screening into general use [52].

The meta study of Wang Memoli et al. pointed out that both classes of IGI bronchoscopy systems—VBN and ENB systems—offer a comparable level of bronchoscopy success [7]. Many drawbacks have been noted for ENB systems, however, most notably hardware cost [12], [21]. VBN systems do not suffer from these many noted drawbacks, but do require an attending technician to help operate [12].

Our work addressed this concern. As our Section III studies demonstrated, a hands-free VBN system, which does not require an attending technician to operate, is feasible. The studies also showed that the system offers substantially faster navigation than currently available technician-assisted VBN systems.

While the physician now has the added responsibility of controlling the system during bronchoscopy, this did not appear to add an unreasonable burden. In reality, the system seems to substantially ease the physician’s burden over that imposed by technician-assisted VBN systems, which require the physician to direct a co-piloting technician. Related to this point, the physician stated she was very pleased with the hands-free system and commented that the user interface worked smoothly. Along this line, well over 90% of the foot-switch commands invoked by the physician involved the single position programmed for “rotate/advance” commands. Hence, the interface appears to be extremely user friendly. We point out, however, that proper training is important.

We also observed that the physician often combined the rotate-flex-advance maneuvers for a given branch into a single maneuver. This proved more convenient in constricted regions, where the physician often preferred to perform several moves at once. Again, our VBN system easily accommodated this approach for two reasons: 1) the system always depicts the complete global navigation plan in the navigation plan tool; and 2) the physician found it straightforward to perform two consecutive foot-pedal commands in line with the maneuvers made. As another point, the base-camp shots of Figs. 5(a) and 6(a) illustrate how the physician generally did not have to try hard in synchronizing the bronchoscope to the system’s directives. In fact, the looseness of the device’s position in Fig. 6(a) probably resulted in the navigation error corrected in the latter part of the figure.

These observations point out the adaptability of the system to many situations. They also illustrate the importance of cooperation and training. So, while the system is able to provide useful guidance information, the physician still is the final arbiter in making decisions and must cooperate when using the system.

The ergonomic study further supported the general acceptability of the system. Regarding this study, while the physician effectively predicted the actual navigation plans followed in the live procedures, 4/7 of the physician’s missed predictions corresponded to navigation plans correctly defined by the system according to the physician. In addition, 5/7 routes that the physician could not properly predict were among the 6 longest (insertion-depth range ≥ 7). Related to this point, we noted situations where neither the automatically planned route nor the physician-suggested maneuvers were followed (e.g., case D, route r4 ). This shows that situations occur where the physician changes the navigation plan “on the fly” during the bronchoscopy procedure. However, even in such situations, having the precomputed navigation plan aids successful guidance. Thus, there is clearly a symbiotic connection between the system’s and the physician’s efficacy, and a useful synergy appears to exist between the system and the human operator.

Regarding system improvements, other physician preferences could be added as options to the navigation-plan computation. For example, certain physicians prefer that all succeeding airways along a route appear situated at the top of the view after every rotation. Accommodating this preference, however, would require relaxing the |α| ≤ 90° restriction on wrist position and allowing for arm/shoulder roll. Finally, as an alternative to the foot switch, we briefly explored a voice-activated interface for controlling the guidance system, but found this approach problematic, given the constant communication and noise in the bronchoscopy suite [42].

V. Conclusion

We proposed a hands-free VBN system for image-guided bronchoscopy. The system automatically produces navigation plans that conform to the physician’s bronchoscopy training while also taking into account the physician’s physical dexterity limits in maneuvering a bronchoscope. During the live procedure, the system enables autonomous hands-free operation by the physician via a simple three-position foot switch, without the need for technician assistance. A guidance strategy, driving the system, translates the navigation plans into intuitive rotate-flex-advance bronchoscope maneuvers and interprets the foot-switch commands, while a set of a dynamically synchronized graphical tools provides salient guidance information. In addition, our system’s unique position-verification mechanism enables a flexible set of operations for verifying bronchoscope position and detecting/correcting erroneous positions.

The system achieved a bronchoscopy navigation success rate =97% in a live human study, comparable to existing technician-assisted VBN systems. The system also appeared to enable significantly faster navigation—on the order of 3 min/route— than the existing systems. Note that a typical bronchoscopic procedure involves three to six sites, with a complete procedure taking 25–45 min, including bronchoscope navigation, site localization, and tissue sampling [10]. Our system shows promise in conceivably enabling a sizable reduction of several minutes per procedure, a major time and cost savings. This time savings is especially striking, given that our system also does not require the extra burden of the attending technician assisting in VBN system operation. While Phantom Study 2 demonstrated this time savings in a randomized two-system test, a randomized patient study is necessary to clearly quantify this benefit in the clinical setting.

In addition, an ergonomic system study—heretofore never done before for an IGI system for bronchoscopy to the best of our knowledge—pointed to the device’s convenience and long-term acceptability for live use by the human operator. Overall, the study found that the typical navigation plan computed by our system presented on average <1 nonideal bronchoscope maneuver per route, implying a very high level of acceptability for our system.

A larger human study would help assess the system’s overall acceptability. In addition, an intriguing possibility is to incorporate the inexpensive sensor-based bronchoscope position-measurement methodology of Cornish or Luo into our current system [25], [26]. Finally, we believe that our general methodologies for navigation planning, guidance, and position verification could also benefit other existing IGI bronchoscopy systems.

Supplementary Material

sup

Acknowledgments

This work was supported in part by the National Cancer Institute of the NIH under Grant R01-CA151433, GrantR01-CA074325, and Grant R44-CA091534.W. E. Higgins and R. Bascom have an identified conflict of interest related to Grant R01-CA151433, which is under management by Penn State and has been reported to the NIH.

M. Amthor, P. Byrnes, R. Cheirsilp, T. Kuhlengel, and X. Zang assisted with the phantom and human studies.

Biographies

graphic file with name nihms879557b1.gif

Rahul Khare received the B.E. in electronics engineering from Mumbai University, Mumbai, India, and the M.S., and Ph.D. degrees in electrical engineering from the Pennsylvania State University, University Park, PA, USA.

He previously held a position at Video Mining, Inc., State College, PA, USA. He is currently a Postdoctoral Fellow in the Sheikh Zayed Institute, Children’s National Medical Center, Washington, DC, USA. His research interests include 3-D medical image processing and visualization, computer vision, machine learning, and pattern recognition.

graphic file with name nihms879557b2.gif

Rebecca Bascom received the A.B. degree in french studies from Harvard University, Cambridge, MA, USA, the M.D. degree from the Oregon Health Sciences University, Portland, OR, USA, and the M.PH. degree from Johns Hopkins University, Baltimore, MD, USA.

She was with the University of Maryland and Johns Hopkins University previously. She is currently a Professor of medicine at the Penn State Milton S. Hershey Medical Center, Hershey, PA, USA. Her research interests include the areas of pulmonary medicine, lung toxicology, and occupational health.

graphic file with name nihms879557b3.gif

William E. Higgins (F’03) received the B.S. degree in electrical engineering from the Massachusetts Institute of Technology, Cambridge, MA, USA, and the M.S. and Ph.D. degrees in electrical engineering from the University of Illinois, Urbana-Champaign, IL, USA.

He has held positions previously at the Honeywell Systems and Research Center, Minneapolis, MN, USA, and the Mayo Clinic, Rochester, MN, USA. He is currently a Distinguished Professor of electrical engineering, computer science and engineering, and bioengineering at the Pennsylvania State University, University Park, PA, USA. His research interests include multidimensional medical image processing, virtual endoscopy, and image-guided intervention systems, with an emphasis on applications for the chest.

Footnotes

Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org.

This paper has supplementary downloadable material available at http://ieeexplore.ieee.org (File size: 43,295 KB).

Contributor Information

Rahul Khare, Department of Electrical Engineering, Pennsylvania State University, and also with Blue Belt Technologies.

Rebecca Bascom, Department of Medicine, Pennsylvania State Hershey Medical Center.

William E. Higgins, Department of Electrical Engineering, the Department of Computer Science and Engineering, and the Department of Bioengineering, Pennsylvania State University, University Park, PA 16802 USA.

References

  • 1.Wang KP, et al. Flexible Bronchoscopy. 2. Cambridge, MA, USA: Blackwell; 2003. [Google Scholar]
  • 2.Sihoe AD, Yim AP. Lung cancer staging. J Surgical Res. 2004 Mar;117(1):92–106. doi: 10.1016/j.jss.2003.11.006. [DOI] [PubMed] [Google Scholar]
  • 3.Rivera MP, Mehta AC. Initial diagnosis of lung cancer: ACCP evidence-based clinical practice guidelines. Chest. 2007 Sep;132(3 Suppl):131S–148S. doi: 10.1378/chest.07-1357. [DOI] [PubMed] [Google Scholar]
  • 4.Dolina MY, et al. Interbronchoscopist variability in endobronchial path selection: A simulation study. Chest. 2008 Apr;133(4):897–905. doi: 10.1378/chest.07-2540. [DOI] [PubMed] [Google Scholar]
  • 5.Minami H, et al. Interbronchoscopist variability in the diagnosis of lung cancer by flexible bronchoscopy. Chest. 1994 Jun;105(2):1658–1662. doi: 10.1378/chest.105.6.1658. [DOI] [PubMed] [Google Scholar]
  • 6.Edell E, Krier-Morrow D. Navigational bronchoscopy: Overview of technology and practical considerations—New current procedural terminology codes effective 2010. Chest. 2010 Feb;137(2):450–454. doi: 10.1378/chest.09-2003. [DOI] [PubMed] [Google Scholar]
  • 7.Wang Memoli J, et al. Meta-analysis of guided bronchoscopy for the evaluation of the pulmonary nodule. Chest. 2012 Aug;142(2):385–393. doi: 10.1378/chest.11-1764. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Merritt SA, et al. Image-guided bronchoscopy for peripheral lung lesions: A phantom study. Chest. 2008 Nov;134(5):1017–1026. doi: 10.1378/chest.08-0603. [DOI] [PubMed] [Google Scholar]
  • 9.Asano F, et al. A virtual bronchoscopic navigation system for pulmonary peripheral lesions. Chest. 2006 Aug;130(2):559–566. doi: 10.1378/chest.130.2.559. [DOI] [PubMed] [Google Scholar]
  • 10.Shinagawa N, et al. Virtual bronchoscopic navigation system shortens the examination time: Feasibility study of virtual bronchoscopic navigation system. Lung Cancer. 2007 May;56(2):201–206. doi: 10.1016/j.lungcan.2006.12.005. [DOI] [PubMed] [Google Scholar]
  • 11.Asano F. Virtual bronchoscopic navigation. Clin Chest Med. 2010 Mar;31(1):75–85. doi: 10.1016/j.ccm.2009.08.007. [DOI] [PubMed] [Google Scholar]
  • 12.Asano F. Practical application of virtual bronchoscopic navigation. In: Mehta A, Jain P, editors. Interventional Bronchoscopy. Vol. 10. New York, NY, USA: Humana Press; 2013. pp. 121–140. (ser. Respiratory Medicine) [Google Scholar]
  • 13.Helferty JP, et al. Computer-based system for the virtual-endoscopic guidance of bronchoscopy. Comput Vis Image Underst. 2007 Oct-Nov;108(1/2):171–187. doi: 10.1016/j.cviu.2006.10.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Higgins WE, et al. 3D CT-video fusion for image-guided bronchoscopy. Comput Med Imaging Graph. 2008 Apr;32(3):159–173. doi: 10.1016/j.compmedimag.2007.11.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Merritt S, et al. Interactive CT-video registration for image-guided bronchoscopy. IEEE Trans Med Imaging. 2013 Aug;32(8):1376–1396. doi: 10.1109/TMI.2013.2252361. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Gibbs J, et al. Optimal procedure planning and guidance system for peripheral bronchoscopy. IEEE Trans Biomed Eng. 2014 Mar;61(3):638–657. doi: 10.1109/TBME.2013.2285627. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Eberhardt R, et al. Lung Point—A new approach to peripheral lesions. J Thoracic Oncol. 2010 Oct;5(10):1559–1563. doi: 10.1097/JTO.0b013e3181e8b308. [DOI] [PubMed] [Google Scholar]
  • 18.Solomon SB, et al. Three-dimensional CT-guided bronchoscopy with a real-time electromagnetic position sensor: A comparison of two image registration methods. Chest. 2000 Dec;118(6):1783–1787. doi: 10.1378/chest.118.6.1783. [DOI] [PubMed] [Google Scholar]
  • 19.Schwarz Y, et al. Electromagnetic navigation during flexible bronchoscopy. Respiration. 2003 Sep-Oct;70(5):515–522. doi: 10.1159/000074210. [DOI] [PubMed] [Google Scholar]
  • 20.Gildea TR, et al. Electromagnetic navigation diagnostic bronchoscopy: A prospective study. Am J Respir Crit Care Med. 2006 Nov 1;174(9):982–989. doi: 10.1164/rccm.200603-344OC. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Gildea T, Cicenia J. Electromagnetic navigation bronchoscopy. In: Mehta A, Jain P, editors. Interventional Bronchoscopy. Vol. 10. New York, NY, USA: Humana Press; 2013. pp. 107–120. (ser. Respiratory Medicine) [Google Scholar]
  • 22.Appelbaum L, et al. Electromagnetic navigation system for CT-guided biopsy of small lesions. Am J Roentgenol. 2011 May;196(5):1194– 1200. doi: 10.2214/AJR.10.5151. [DOI] [PubMed] [Google Scholar]
  • 23.Mori K, et al. Compensation of electromagnetic tracking system using an optical tracker and its application to bronchoscopy navigation system. Proc SPIE. 2007;6509:65090M-1–65090M-12. [Google Scholar]
  • 24.Soper TD, et al. In vivo validation of a hybrid tracking system for navigation of an ultrathin bronchoscope within peripheral airways. IEEE Trans Biomed Eng. 2010 Mar;57(3):736–745. doi: 10.1109/TBME.2009.2034733. [DOI] [PubMed] [Google Scholar]
  • 25.Cornish DC, Higgins WE. Bronchoscopy guidance system based on bronchoscope-motion measurements. Proc SPIE. 8316:83161G-1–83161G-11. [Google Scholar]
  • 26.Luo X, et al. Externally navigated bronchoscopy using 2-D motion sensors: Dynamic phantom validation. IEEE Trans Med Imaging. 2013 Oct;32(10):1745–1764. doi: 10.1109/TMI.2013.2263152. [DOI] [PubMed] [Google Scholar]
  • 27.Vining DJ, et al. Virtual bronchoscopy: Relationships of virtual reality endobronchial simulations to actual bronchoscopic findings. Chest. 1996 Feb;109(2):549–553. doi: 10.1378/chest.109.2.549. [DOI] [PubMed] [Google Scholar]
  • 28.Higgins WE, et al. Virtual bronchoscopy for 3D pulmonary image assessment: State of the art and future needs. Radiographics. 1998 May-Jun;18(3):761–778. doi: 10.1148/radiographics.18.3.9599397. [DOI] [PubMed] [Google Scholar]
  • 29.Haponik EF, et al. Virtual bronchoscopy. Clinics Chest Med. 1999 Mar;20(1):201–217. doi: 10.1016/s0272-5231(05)70135-0. [DOI] [PubMed] [Google Scholar]
  • 30.Bricault I, et al. Registration of real and CT-derived virtual bronchoscopic images to assist transbronchial biopsy. IEEE Trans Medical Imaging. 1998 Oct;17(5):703–714. doi: 10.1109/42.736022. [DOI] [PubMed] [Google Scholar]
  • 31.McAdams HP, et al. Virtual bronchoscopy for directing transbronchial needle aspiration of hilar and mediastinal lymph nodes: A pilot study. Am J Roentgenol. 1998 May;170(5):1361–1364. doi: 10.2214/ajr.170.5.9574616. [DOI] [PubMed] [Google Scholar]
  • 32.Hopper K, et al. Transbronchial biopsy with virtual CT bronchoscopy and nodal highlighting. Radiology. 2001 Nov;221(2):531–536. doi: 10.1148/radiol.2211001585. [DOI] [PubMed] [Google Scholar]
  • 33.“Bronchoscopy step-by-step,” an Electronic On-Line Multimedia Slide Presentation. [Online] 2004 Available: http://www.bronchoscopy.org.
  • 34.Nadeem SM. “Fiberoptic bronchoscopy: The technique,” educational material from Committee for European Education in Anesthesiology. [Online] 2009 Available: http://www.euroviane.net.
  • 35.Mori K, et al. Automated anatomical labeling of the bronchial branch and its application to the virtual bronchoscopy system. IEEE Trans Med Imaging. 2000 Feb;19(2):103–114. doi: 10.1109/42.836370. [DOI] [PubMed] [Google Scholar]
  • 36.Kiraly AP, et al. 3D path planning for virtual bronchoscopy. IEEE Trans Med Imaging. 2004 Nov;23(11):1365–1379. doi: 10.1109/TMI.2004.829332. [DOI] [PubMed] [Google Scholar]
  • 37.Kiraly AP, et al. 3D human airway segmentation methods for clinical virtual bronchoscopy. Acad Radiol. 2002 Oct;9(10):1153–1168. doi: 10.1016/s1076-6332(03)80517-2. [DOI] [PubMed] [Google Scholar]
  • 38.Lu K, Higgins WE. Interactive segmentation based on the live wire for 3D CT chest image analysis. Int J Comput Assisted Radiol Surgery. 2007 Dec;2(3/4):151–167. [Google Scholar]
  • 39.Yu KC, et al. System for the analysis and visualization of large 3D anatomical trees. Comput Biol Med. 2007 Dec;37(12):1802–1820. doi: 10.1016/j.compbiomed.2007.06.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Gibbs JD, et al. 3D MDCT-based system for planning peripheral bronchoscopic procedures. Comput Biol Med. 2009 Mar;39(3):266–279. doi: 10.1016/j.compbiomed.2008.12.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Graham MW, et al. Robust 3D airway-tree segmentation for image-guided peripheral bronchoscopy. IEEE Trans Med Imaging. 2010 Apr;29(4):982–997. doi: 10.1109/TMI.2009.2035813. [DOI] [PubMed] [Google Scholar]
  • 42.Khare R. PhD dissertation. Pennsylvania State Univ; University Park, PA, USA: 2012. Global registration for image-guided bronchoscopy. [Google Scholar]
  • 43.Khare R, Higgins WE. Toward image-based global registration for bronchoscopy guidance. Proc SPIE. 2010 Feb;7625:762510. doi: 10.1117/12.839479. [DOI] [Google Scholar]
  • 44.Yu KC, et al. Image-based reporting for bronchoscopy. J Digital Imaging. 2010 Feb;23(1):39–50. doi: 10.1007/s10278-008-9170-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Graham MW, et al. Computer-based route-definition system for peripheral bronchoscopy. J Digital Imaging. 2012 Apr;25(2):307–317. doi: 10.1007/s10278-011-9433-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.Amthor M. Bachelor’s thesis. Dept. Elect. Eng., Pennsylvania State Univ; University Park, PA, USA: 2013. Quantitative analysis of the human airway tree using high resolution MDCT data. [Google Scholar]
  • 47.Kukuk M. Modeling the internal and external constraints of a flexible endoscope for calculating its workspace: application in transbronchial needle aspiration guidance. Proc SPIE. 2002;4681:539–550. [Google Scholar]
  • 48.Shinagawa N, et al. Factors related to diagnostic sensitivity using an ultrathin bronchoscope under CT guidance. Chest. 2007 Feb;131(2):549–553. doi: 10.1378/chest.06-0786. [DOI] [PubMed] [Google Scholar]
  • 49.Camarillo D, et al. Robotic technology in surgery: Past, present, and future. Am J Surgery. 2004 Oct;188(4):2S–15S. doi: 10.1016/j.amjsurg.2004.08.025. [DOI] [PubMed] [Google Scholar]
  • 50.DiMaio S, et al. Challenges in image-guided therapy system design. NeuroImage. 2007;37:S144–S151. doi: 10.1016/j.neuroimage.2007.04.026. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 51.Manzey D, et al. Image-guided navigation: The surgeon’s perspective on performance consequences and human factors issues. Int J Med Robot Comput Assist Surg. 2009 May;5(3):297–308. doi: 10.1002/rcs.261. [DOI] [PubMed] [Google Scholar]
  • 52.Aberle D, et al. Reduced lung-cancer mortality with low-dose computed tomographic screening. New England J Med. 2011;365(5):395– 409. doi: 10.1056/NEJMoa1102873. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

sup

RESOURCES