Dual-mode laparoscopic fluorescence image-guided surgery using a single camera

Abstract

Iatrogenic nerve damage is a leading cause of morbidity associated with many common surgical procedures. Complications arising from these injuries may result in loss of function and/or sensation, muscle atrophy, and chronic neuropathy. Fluorescence image-guided surgery offers a potential solution for avoiding intraoperative nerve damage by highlighting nerves that are otherwise difficult to visualize. In this work we present the development of a single camera, dual-mode laparoscope that provides near simultaneous display of white-light and fluorescence images of nerves. The capability of the instrumentation is demonstrated through imaging several types of in situ rat nerves via a nerve specific contrast agent. Full color white light and high brightness fluorescence images and video of nerves as small as 100 µm in diameter are presented.

1. Introduction

Iatrogenic nerve damage is a leading cause of morbidity associated with surgical procedures, including prostatectomy [1,2], coronary artery bypass graft [3,4], thyroidectomy [5,6], lumpectomy and mastectomy [7,8]. Complications arising from these injuries are dependent on the severity and location of the nerve injury but may result in loss of function and/or sensation, muscle atrophy, and chronic neuropathy [9]. The exact cause of nerve injury during laparoscopic and open surgical procedures is highly variable. However, inadvertent surgical damage due to poor visibility of nerves in comparison to surrounding tissue, or simply the unfortunate necessity due to proximity of the nerve to target structures remains a risk. Currently employed nerve sparing techniques rely primarily upon anatomical landmark identification or use of intraoperative nerve electrical stimulation devices to verify stimulation of innervated muscles or organs [10,11]. Surgical procedures are typically performed without any form of image guidance as the available technologies lack specificity needed to provide nerve-specific imaging [12].

Optical imaging allows for both microscopic [13–15] and macroscopic [16–19] visualization of nerves during surgery. One method for intraoperatively identifying nerves and other critical structures is through the use of fluorescence image guided surgery (FIGS), which in recent years has undergone substantial growth leading to clinical translation [20–24] and commercialization. The surgical technique consists of optical contrast agents coupled with optical imaging hardware to provide visualization and increased contrast of otherwise indistinguishable anatomical features [25–28]. A range of targeted contrast agents have been used for tumor margin delineation [29] and visualization of nerves [16–18], while non-specific agents such as ICG have been used for sentinel lymph node mapping [21,22,26] and for vascular imaging [23,30].

Presently, optical imaging hardware for FIGS is undergoing rapid development in functionality and miniaturization [31–33]. Minimally invasive surgery (MIS) is gaining prevalence because it offers the benefits of small incisions, less pain, faster recovery, and fewer post-operative complications. Until recently laparoscopic surgeries were performed using white light illumination and color digital cameras, but have recently been enhanced with the capability of fluorescence imaging. Conventional laparoscopes have been outfitted with filtered illumination and filtered cameras to provide fluorescence guidance in pre-clinical [34,35] and clinical procedures [22,23]. Additionally, commercial MIS systems are becoming available with integrated fluorescence imaging capabilities (e.g., Karl Storz D-Light, O₂view Artemis, and Intuitive Surgical Firefly [23,36,37]).

Standard laparoscopes (10 mm or 5 mm diameter) have a large field of view (70-135 degrees) and small collection aperture (typically F/8 to F/12). FIGS imaging with laparoscopes faces specific challenges due to low collection efficiencies and the requirement for compact and lightweight equipment. The need for a lightweight system precludes the use of sensitive, scientific grade cameras which typically include large detectors and cooling circuitry, or the use of two simultaneous cameras split with a dichroic beam splitter or similar methods [20,26,31]. However a recent approach used an image-preserving fiber bundle to relay the laparoscope output image onto a sensitive electron-multiplying CCD [38], rather than mounting the detector directly on the laparoscope. In open surgical procedures, where the surgeon can directly view the surgical area, a single channel fluorescence-only FIGS system may be sufficient. However in MIS, obtaining white light video is a requisite feature of any laparoscopic system. Therefore, MIS FIGS should ideally perform dual-mode imaging with a single camera. Presently, dual-mode laparoscopic imaging is typically achieved by manual switching between a white light and fluorescence image [22,23,34,36] or by allowing partial leakage of the white light through the emission filter [35]. Several hybrid approaches to simultaneous multi-spectral imaging have been demonstrated for surgical imaging, incorporating multiple cameras and dichroic mirrors [39], hybrid color cameras with custom filter mosaics [40], fast liquid crystal tunable filters [41] and modulated light sources coupled to the camera acquisition [42]. Thus far these have been restricted to reflected light only.

In this work we present a simple, low-cost, and lightweight system that provides near simultaneous acquisition of white-light and fluorescence video via time multiplexing. The system uses a single camera which allows for easy registration of the two channels for optional overlay and is compatible with any standard laparoscope. A custom built illumination module is electronically synchronized to a color camera that captures alternating white light and fluorescence images at 30 frames per second (fps). This architecture allows for the maximum display frame rate and adaptability of the system to any camera for dual mode imaging. The capability of the system is demonstrated in a rodent model by imaging several types of nerves in situ, via a nerve-specific fluorophore, GE3082, previously described [17].

2. Methods

2.1. Instrumentation

The imaging system (Fig. 1 ) consists of three primary modules: laparoscope, illumination, and control and acquisition. The laparoscope module comprises a standard 10 mm zero degree surgical laparoscope with 70 degree field of view (T1000 Linvatec, Largo, FL), a 4 mm diameter, 1800 mm long laparoscope light guide (Medit Inc., Winnipeg, Canada), a 35 mm video coupler (MVC-35 Medit Inc.), a compact 90 gram, 659x494 pixel GigE color camera (acA640-90gc Basler, Ahrensburg, Germany), and a 405 nm blocking filter (BLP01-405R Semrock, Rochester, NY), which has >97% transmission from 420 to 800 nm. The 1/3” format sensor provides high sensitivity with 7.4 µm pixel size and adequate field of view (40 degrees out of 70 degrees passed by the laparoscope). The video coupler interfaces with the eyepiece of the laparoscope providing an interchangeable set up for different types of laparoscopes. The 405 nm filter is secured directly in front of the image sensor with a c-mount retaining ring. The camera threads onto the video coupler directly. The illumination light guide connects directly to each laparoscope with the appropriate mechanical coupler.

Fig. 1.

Single camera, dual-mode laparoscopic FIGS instrument. (a) Photograph of the laparoscope, coupler, camera with filter, and light guide. (b) Schematic of the illumination path. The LED and 405 nm laser are collimated and coupled into the light guide with aspheric lenses and combined with a dichroic mirror. A longpass filter in the LED path filters the white light spectrum below 450 nm.

Illumination for white light and fluorescence imaging is coupled into a single light guide using conventional optics. The light coupling module (Fig. 1(b)) consists of two 32 mm aspheric lenses (ACL4532, Thorlabs, Newton, NJ) to collimate a white light LED (XPGWHT-L1-0000-00H53, Cree Inc., Durham, NC) and a 500 mW, 405 nm blue laser diode coupled into a multimode fiber with 400 μm diameter (Shanghai Laser & Optics Century Co., Ltd., China). The LED spectrum is filtered with a 450 nm long pass filter (NT49-819, Edmund Optics, Barrington, NJ) to minimize excitation of the fluorescent agent, while maintaining the white light color spectrum. The LED and laser are combined with a 425 nm dichroic mirror (DMLP425R, Thorlabs). The combined illumination is coupled into the light guide with a third 32 mm aspheric lens (ACL4532, Thorlabs). The maximum irradiance from the LED and 405 nm laser are 2.0 mW/cm² and 7.3 mW/cm², respectively, at 25 mm from the tip of the laparoscope. These laser powers represent class 3B operation when compared to the ANSI and IEC laser safety standards (ANZI Z136.1-2007, IEC 60825-1 2007) when eye exposure occurs at the laparoscope tip. When accidentally viewed from over 0.5 m, the diverging illumination hazard is well within class 1.

Electronic synchronization between the camera and illumination sources provides nearly simultaneous display of white light and fluorescence frames. Synchronization is achieved by triggering a 4 channel digital pulse generator (DG535, Stanford Research Systems, Inc., Sunnyvale, CA) with the output line of the GigE camera, which generates a 5V signal when the camera exposure is active. The camera acquisition is stopped and started from the PC; thus the illumination is automatically synchronized. The exposure active pulse is used to synchronize TTL pulses with pre-set delay and pulse width from the digital pulse generator, as shown in Fig. 2 . The stop and start time of each pulse is precisely aligned to ensure no bleed through to the opposing frames.

Fig. 2.

Timing diagrams indicating synchronization of the camera and light sources. The camera sync output (black) is used to drive the laser (red) and LED (blue) modulation via a digital pulse generator. All waveforms are 5V in amplitude.

The LED driver (D1B, Thorlabs) accommodates TTL input and provides LED illumination power control. The 405 nm laser is modulated by a TTL input and power controlled by the laser driver. Raw interleaved camera frames are acquired using a work station (Hewlett-Packard, xw8400, Palo Alto, CA) with custom image acquisition software written in C++. Images were adjusted for contrast and brightness in ImageJ (National Institutes of Health, Bethesda, MD). Since the images are obtained with a color camera, both channels represent true-color images of light reflected or emitted from the sample. This in contrast to conventional imaging, where fluorescence images are typically rendered using a false-color intensity display.

2.2. Fluorescent contrast agent

Detailed optical characterization, in vivo dosing and kinetics studies for GE3082 has been previously published [17]. However, the optical properties of GE3082 were reevaluated here in various solvents to prevent inconsistencies due to instrumentation (Table 1 ). Briefly, a 10 mM stock solution of GE3082 was prepared in anhydrous dimethylsulfoxide (DMSO) to ensure complete dissolution of the fluorophore. Subsequent aliquots of the stock solution were taken to prepare 10 μM solutions of GE3082 in the following solvents: DMSO, water, methanol, and a selected intravenous (IV) formulation (35% propylene glycol, 35% polyethylene glycol-300, 18% 2-hydroxypropyl-β-cyclodextrin, 11.5% distilled water, and 0.5% DMSO).

Table 1. Optical characterization of GE3082.

Solvent	ε (M−1 cm−1)	Exc (nm)	Em (nm)	QY (%)
DMSO	37200	417	623	1.7
MeOH	41900	395	592	1.0
Water	19356	380	578	0.8
IV formulation*	32898	396	594	1.5

*Contained 35% propylene glycol, 35% polyethylene glycol-300, 18% 2-hydroxypropyl-β-cyclodextrin, 11.5% distilled water, and 0.5% DMSO; ε = molar extinction coefficient; Exc = excitation maximum wavelength; Em = emission peak wavelength; QY = quantum yield.

Absorbance spectra were taken using a Lambda 20 UV/Vis spectrometer (Perkin Elmer, Waltham, MA). The wavelength of maximum absorbance was then used as the excitation wavelength for the collection of the fluorescence emission spectra on a steady state spectrofluorometer (Photon Technology International, Birmingham, NJ). The molar extinction coefficient (ε) for GE3082 was calculated at the maximum excitation wavelength corresponding to each solvent, using Beer-Lambert’s law. The quantum yield (QY) of GE3082 in each solvent was measured in comparison to the fluorescence emission of a known standard (Coumarin-6; QY = 78%)) [43] using the single-point method [44]. An enhancement of fluorescence emission was observed upon binding to purified native myelin basic protein (MBP), a constituent of myelinated nerves (Fig. 3 ). Here, 10 μM of GE3082 was diluted in CHAPS (3[(3-cholamidopropyl)dimethylammonio]-propanesulfonic acid) buffer. Fluorescence emission spectra (405 nm excitation) were measured from the solution before and after the addition of 1.6 μM of MBP, resulting in an average enhancement factor of 17 across the spectrum, upon introduction of the protein.

Fig. 3.

Fluorescence spectra of the contrast agent GE3082 taken before (blue) and after (red) addition of native myelin basic protein (MBP), its biological target. The agent demonstrates increased fluorescence intensity upon binding to MBP. The average of the enhancement factor (green triangles) across this spectral range is 17.0 ± 0.72 for the specified concentrations of agent (10 μM) and protein (1.6 μM).

2.3. Animal preparation and surgical imaging

All procedures were approved by the Institutional Animal Care and Use Committee (IACUC) at GE Global Research. Male Sprague-Dawley rats ranging in body weight from 250 to 300 g were purchased from Charles River Laboratories (Wilmington, MA) and housed at 22-23°C on a 12-hour light/dark cycle. Rats were maintained on rodent chow (LabDiet 5001, Framingham, MA) and water ad libitum. On the day of the experiment, rats were anesthetized using 2-4% isofluorane and given a single tail vein injection of either GE3082 (11 mg/kg) in formulation buffer (35% propylene glycol, 35% polyethylene glycol 300, 18% 2-hydroxypropyl β-cyclodextrin, 11.5% sterile water, 0.5% dimethylsulfoxide) or formulation buffer alone as a control. The rats were then returned to the home cage until the designated time point for imaging. Four hours post-injection, the rats were euthanized using compressed CO₂ gas, and major nerves were exposed and imaged. In addition, a small incision was made below the sternum allowing for passage of the laparoscope into the closed thoracic cavity for phrenic nerve imaging. Five different locations were imaged in the experimental rat. The range of locations demonstrates the extent of nerve labeling, taking into account nerves of differing sizes and myelination, and thus the overall imaging capability. Only the sciatic nerve was imaged in the control animal, since it is heavily myelinated. After the procedure was complete, the same locations were imaged using a fluorescence microscope (SteREO Lumar, Carl Zeiss, Thornwood, NJ) to measure the physical dimensions of each nerve.

3. Results

To test the imaging properties of the laparoscope, a USAF resolution target (Edmund Optics, NT38-710) was first used as a sample to characterize the spatial resolution of the assembled laparoscope. A white light external lamp was used to avoid specular reflection from the target protective lamination. On axis and edge images where acquired as shown in Fig. 4 . At a working distance of 25 mm from the tip of the laparoscope, 7.13 lp/mm (line pairs per mm) were resolved on axis and at the edge of the FOV. At 50 mm working distance, 5.04 lp/mm were resolved. These correspond to a spatial resolution of ~70 μm and ~100 μm respectively.

Fig. 4.

Image of USAF resolution chart taken on axis at a working distance of 25 mm, demonstrating the spatial resolution of the instrument. Group number 2, element 6 can be easily resolved, corresponding to 7.13 lp/mm, or ~70 μm.

During imaging, ambient room lighting was minimized in order to prevent light leakage. The camera gain, laparoscope focus, and power of the LED and laser were adjusted to maximize image quality. Several myelinated nerves, such as the sciatic, brachial, median, vagus, and phrenic nerves were imaged using white light and fluorescence, as shown in Fig. 5 . The identity of each nerve was verified under white light conditions. Fluorescence was clearly visible in all nerves imaged, allowing for easy visualization of major nerve trunks and their branches. Fluorescence was also observed in adipose tissue, while the surrounding muscle tissue was dark. In Fig. 5(b) the major trunk of the brachial plexus nerve spans the length of the image. Fluorescence signal from smaller nerves such as the suprascapular (horizontal arrow in Fig. 5(b)) and branches of the phrenic, radial, ulnar, and median nerves are also visible (yellow arrow in Fig. 5(b)). Moreover, distinct borders of the nerves were clearly distinguishable from adjacent tissue and blood vessels in the fluorescence images. For example, in Fig. 5(d), the vagus nerve runs parallel to the carotid artery and nearby trachea. In this white light image, the borders of the vagus were not easily defined along the entire length of the nerve in comparison to other adjacent tissue types. In contrast, the borders of the vagus are readily seen against the adjacent carotid artery and nearby trachea and thyroid gland in the corresponding fluorescence image, Fig. 5(d).

Fig. 5.

Laparoscopic surgery of fluorescently labeled nerves in rodent model, shown under white light (left column) and fluorescence (right column) obtained with the single camera, dual-mode laparoscopic instrument. Both channels are displayed in true color. All scale bars = 1 mm. Red arrows indicate the primary nerves imaged, which include the (a) sciatic, (b) brachial, (c) median, (d) vagus, and (e) phrenic nerves. The yellow arrow (b) shows a branch of the phrenic nerve, with ~100 µm diameter, and branches of the radial, ulnar, and median nerves in close proximity.

Similar findings were observed when imaging the phrenic nerve through a small incision in a closed thoracic cavity, more closely mimicking a minimally invasive surgery. In Fig. 5(e) the phrenic nerve is obscured under white light by attached parietal pleura. However, in the corresponding fluorescence image, distinct fluorescence of the phrenic nerve could be clearly seen along its entire length, regardless of the obscuring tissue seen in the white light image. Overall, these findings demonstrate visualization of nerves as small as 100 µm in diameter with high fidelity both in open and minimally invasive conditions.

The composite image acquisition rate was 30 fps, resulting in an effective frame rate of 15 fps per channel, which is high enough for motion-artifact and blur-free imaging. A video sequence of the phrenic nerve (Fig. 6 ) shows high fidelity imaging in recorded in real-time. In the video the heart, lungs and sternum are also visible. Although there is a single frame lag between the two channels, the small delay (33 ms) is not readily apparent.

Fig. 6.

Single-frame excerpt from video recordings of simultaneous white light (left) and fluorescence (right) imaging of the rat phrenic nerve (Media 1 ^{(2.3MB, AVI)} ).

The sciatic nerve of a control rat (no fluorophore administered) was imaged under the same illumination, imaging distance, and camera settings as the experimental rat, as shown in Fig. 7 . A histogram of the image intensity shows no measurable signal in the fluorescence channel, indicating that autofluorescence is negligible under these measurement conditions.

Fig. 7.

Results of control experiment, highlighting the sciatic nerve under (a) white light and (b) fluorescence. The inset in (b) is the image intensity histogram, showing no signal in the fluorescence channel, in the absence of injected dye. Scale bar ~1 mm.

4. Discussion

Reduction and prevention of nerve damage in surgery is an unmet clinical need requiring innovation in contrast agents as well as imaging hardware. A novel system incorporating both a contrast agent and fluorescence imaging instrumentation has been demonstrated. The instrumentation provided simultaneous real-time display of reflected white light and fluorescence images of nerves as small as 100 µm. The use of a single color camera streamlined the design, allowed for a form factor comparable to conventional laparoscopes, and provided inherent registration between the two channels. Factors such as the excitation and emission spectra, utilization in a closed environment (as is present in minimally invasive surgery), advancement of light sources and available camera technology have made this single-camera, dual-mode imaging solution feasible using a combination of standard, low cost equipment.

The promising imaging results obtained with the system using a visible fluorophore suggest that absorption and emission in the visible spectrum may allow for high fidelity imaging in an intraoperative setting. Here the agent can be excited at the edge of the normal white spectrum (405 nm laser) and the emission can be collected over the same waveband as the white light images. The camera is only filtered by a single 405nm laser line filter, thus allowing un-compromised white light reflectance imaging using an LED. Furthermore, the high brightness of the contrast agent affords high SNR images with minimal excitation power.

Recently, several works have demonstrated the advantages of contrast agents for FIGS in the NIR spectrum, because of its separation from the visible spectrum (white light channel), reduced autofluorescence, and prospects for deeper imaging penetration [45,46]. However, in controlled lighting environments such as MIS or with specialized room lighting, an agent excited at the blue end of the spectrum may also be advantageous. In particular, a highly specific, targeted fluorophore with a large Stokes shift can provide a high signal-to-background ratio despite operating in the visible spectrum. Excitation at 405 nm allows for laser line rejection with minimal impact on white light image quality, and emission in the visible benefits from large detector quantum efficiency within this range. Furthermore, this approach is complementary to NIR imaging, and does not interfere with NIR fluorescence if multi-channel molecular imaging is desired [17].

While many of these benefits result from the optical properties of the present fluorophore, the single-camera dual-mode design is adaptable to other contrast agents. Adaptation to other dyes requires merely changing the laser source and corresponding rejection filter to match the excitation wavelength. A notch filter can be used for laser rejection while transmitting the remainder of the visible spectrum. Multiband dichroic mirrors can be used to combine the white light and laser source without excessive loss of light.

Several fluorescent nerve-highlighting contrast agents have been described recently, including non-targeted NIR fluorophores that are injected directly at the nerve of interest [19], a non-toxic fragment of tetanus toxin for labeling retrograde transport in nerves [47], and peptides that bind to the connective tissue in the nerve epineurium and endoneurium [16]. GE3082, and a related analog we reported on previously (GE3111) [18] are small molecule fluorophores capable of crossing the blood-nerve and blood-brain barriers, a key requirement for systemic administration of targeted nerve contrast agents. Small molecules are generally less costly to produce than peptides and protein fragments, and they may be modified by chemistry to confer more desirable properties, such as stronger binding to a protein target or improved in vivo pharmacokinetics.

However, most molecules that cross the blood-nerve and blood-brain barriers have high lipid solubility, resulting in non-specific partitioning to adipose tissue. GE3082 appears to exhibit higher fluorescence intensity in adipose tissue than in nerves. One possible explanation for this observation is the dependence of agent optical property in local environment. A related analog, GE3111, exhibited an increase in quantum yield when measured in a non-polar solvent, such as olive oil [17].

For clinical translation of myelin-targeting FIGS, better differentiation of nerve from adipose tissue fluorescence may be necessary. Using a commercial multispectral instrument, we have previously shown that GE3082 nerve fluorescence appeared a different color than its adipose tissue fluorescence [18]. Moreover, nerve and adipose tissue appear quite different structurally. Therefore, future initiatives will explore multispectral detection and shape-based segmentation/image processing to better differentiate nerve from adipose tissue fluorescence.