Real-time three-dimensional Fourier-domain optical coherence tomography video image guided microsurgeries

Abstract.

The authors describe the development of an ultrafast three-dimensional (3D) optical coherence tomography (OCT) imaging system that provides real-time intraoperative video images of the surgical site to assist surgeons during microsurgical procedures. This system is based on a full-range complex conjugate free Fourier-domain OCT (FD-OCT). The system was built in a CPU-GPU heterogeneous computing architecture capable of video OCT image processing. The system displays at a maximum speed of $10\text{\hspace{0.17em}\hspace{0.17em}volume}/s$ for an image volume size of $160\times 80\times 1024$ $(X\times Y\times Z)$ pixels. We have used this system to visualize and guide two prototypical microsurgical maneuvers: microvascular anastomosis of the rat femoral artery and ultramicrovascular isolation of the retinal arterioles of the bovine retina. Our preliminary experiments using 3D-OCT-guided microvascular anastomosis showed optimal visualization of the rat femoral artery ($\text{diameter}<0.8\mathrm{mm}$), instruments, and suture material. Real-time intraoperative guidance helped facilitate precise suture placement due to optimized views of the vessel wall during anastomosis. Using the bovine retina as a model system, we have performed “ultra microvascular” feasibility studies by guiding handheld surgical micro-instruments to isolate retinal arterioles ($\text{diameter}\sim 0.1\mathrm{mm}$). Isolation of the microvessels was confirmed by successfully passing a suture beneath the vessel in the 3D imaging environment.

1. Introduction

Microsurgical working spaces are typically small, confined, and not fully accessible. In the case of delicate structures, such as the retina and its microvasculature, it poses a very challenging surgical environment for surgeons. Access and visualization difficulties are further magnified by the small dimensions, on the order of tens or hundreds of microns, and the fragility of these tissues. Performance of microsurgery requires extensive training as well as innate fine motor control. Microsurgical technique currently involves visualization of the surgical target in the ($X$, $Y$) plane on the tissue surface using a high quality binocular surgical microscope.¹ The ability to view critical parts of a tissue and to work in micron proximity to fragile tissue surfaces requires excellent visibility, precision micro-instruments, and a highly skilled surgeon. At present the surgeon must function within the physiological limits of human depth perception to visualize targets, steadily guide microsurgical tools, and execute all surgical objectives.² These directed surgical maneuvers must occur simultaneously with minimization of surgical risk and expeditious resolution of any resulting complications. The main objective of this work is directed at providing microsurgeons with real-time ($X$, $Z$) or ($Y$, $Z$), depth-resolved video views that will enhance their free-hand ability to achieve surgical objectives, diminish surgical risk, and improve outcomes. Optical coherence tomography (OCT) has been implemented and studied as a novel method of microsurgical guidance.³ Compared to other image-guiding modalities such as MRI, CT, and ultrasound, OCT has the potential to be faster, compact, and high-resolution.

Since its introduction into the clinical field of ophthalmology OCT scanning has emerged as one of the most utilized diagnostic applications in the field.⁴ Over the past decade numerous technological breakthroughs have been made; OCT now offers unparalleled resolution in real time. Progress in OCT imaging has come at both the system and component levels.^4–23 This progress includes but is not limited to several advances: fabrication of new, broader light sources for achieving finer resolution (wavelength-swept source, ultra high-speed pulsed laser); building of faster spectrometers; development of compact two-dimensional lateral scanning tools; design for miniaturized probes; manipulation and processing of various noises on the obtained images; optical control based on the OCT system; adaptive morphological imaging (polarization-sensitive OCT, Doppler OCT); functional imaging (blood flow, oxymetry, cell functions); additional fields of applications (clinical and non-clinical fields); and image processing for obtained images.

There have been attempts to use OCT as an interventional imaging tool and to guide neurosurgical procedures in small rodent models.²⁴ Recently our group has presented the development of “smart” microsurgical tool systems based on common-path OCT that provide surface topology and motion compensation^25,26 allowing imaging and sensing in the axial direction of the microsurgical tool. By controlling tool axial motion these novel systems are able to provide surgical accuracy beyond current human physical ability.²⁷ Most OCT systems currently available exhibit limited image processing and possess display speeds that hinder delivery of real-time OCT video images. High-speed real-time OCT is an essential feature required to assist in microsurgery since it is critical that image acquisition be able to track tool tips relative to the tissue surface and simultaneously monitor involuntary tissue target motion (e.g., motion secondary to respiratory effort, cardiac activity or vascular pulsation).²⁸ These considerations are especially critical in delicate operations, such as in the cerebral cortex during neurosurgery,²⁹ the retina during retinal microsurgery,³⁰ and micro-vascular anastomosis of blood vessels $<1\mathrm{mm}$ in diameter frequently required in reconstructive surgery.³¹

2. Real-Time 4D Optical Coherence Tomography

We performed the experiment using our in-house-developed FD-OCT system, as shown in Fig. 1. The initial system was assembled a few years ago, and we have been continuously improving the hardware and software of the system for applications in interventional imaging. The system uses an in-house custom designed and built high-speed spectrometer that uses a 12 bit dual-line CMOS line-scan camera (Sprint spL2048-140k, Basler AG, Germany). The camera was set to work at 1024-pixel mode by selecting the area-of-interest (AOI) where the light source spectrum was fully covered. The minimum line period of the camera was set to 7.8 μs, which corresponds to a maximum line rate of 128k A-scan/s. A superluminescent diode (SLED) (${\lambda }_0=825\mathrm{nm}$, $\mathrm{\Delta }\lambda =70\mathrm{nm}$, Superlum, Ireland) was used as the light source, which provided a measured axial resolution of approximately 5.5 μm in air. We picked 825 nm as a center wavelength for several reasons: 1. this wavelength gives reasonable penetration depth; 2. the shorter wavelength gives reasonably higher axial and transverse resolution for a given bandwidth; and 3. 825 nm is very close to the crossover wavelength for oxygenated hemoglobin absorption and it is not the focus of this effort, it will also allow us to perform standard oximetry as an added capability.³² The image scanning was implemented by a pair of galvanometer mirrors driven by a function generator and synchronized to a high-speed frame grabber from National Instruments, PCIE-1429. To provide a sufficient view of the relevant surgical site in retina, we chose an imaging volume of $3\mathrm{mm}(X)\times 3\mathrm{mm}(Y)$ lateral and $3\mathrm{mm}(Z)$.

Fig. 1.

System configuration: CMOS, CMOS line scan camera; G, grating; L1, L2, L3, L4 achromatic collimators; SL, scanning lens; DCL, dispersion compensation lens; C, $50:50$ broadband fiber coupler; He-Ne, pigtailed guiding light source; M, reference mirror; PC, polarization controller; CL, camera link cable; CTRL, galvanometer control signal; GVS, galvanometer pairs (for simplicity only the first galvanometer is illustrated); GUI, graphics user interface; OS, operation stage.

To remove complex conjugate images that are detrimental to surgical guidance, we have implemented a full-range complex mode operation by applying phase modulations to B-scan 2D interferogram frames by slightly displacing the probe beam off the first galvanometer’s pivoting point (here only the first galvanometer is illustrated in Fig. 1).^33,34 Depending on the frame rate and the image size, the beam offset from the galvanometer pivoting point was changed to ensure that $\mathrm{pi}/2$ phase shift was applied across each B-mode image. For the scanning range of $3\mathrm{mm}(X)\times 3\mathrm{mm}(Y)$ lateral and $3\mathrm{mm}(Z)$, the galvanometer scanning angle was approximately 7 deg and the beam offset was approximately 1.6 mm.³⁴

Our dual-GPUs architecture assigns different tasks to each GPU and has advantages in terms of both imaging system stability and software engineering perspectives to efficiently process data and render images.³⁵ For the real-time 4D imaging mode, the volume rendering is only conducted when a complete C-scan is ready, while B-scan frame processing is running continuously. Therefore, if the signal processing and the visualization are performed on the same GPU, competition for GPU resource will happen when the volume rendering starts while the B-scan processing is still going on, which could result in instability for both tasks. Simply dividing the data and streaming them to two GPUs for processing and displaying cannot solve this issue. Therefore, assigning different computing tasks to different GPUs makes the entire system more stable and consistent.

To host two GPU cards, we chose a quad-core Dell T7500 workstation to control the OCT and process OCT video images. The workstation hosted the frame grabber (PCIE-x4 interface), DAQ card (PCI interface), GPU-1 and GPU-2 (both PCIE-x16 interfaces), all on the same motherboard. The dual GPU set-up was used to increase overall OCT image processing and rendering speed.³⁵ In this platform the GPU-1 (NVIDIA GeForce GTX 580) with 512 stream processors, 1.59 GHz processor clock, and 1.5 GByte graphics memory was dedicated for raw data processing of B-scan frames; the GPU-2 (NVIDIA GeForce GTS450) with 192 stream processors, 1.76 GHz processor clock and 1.0 GBytes graphics memory) was dedicated for volume rendering and display of the complete C-scan data processed by the GPU-1. The GPUs were programmed through NVIDIA’s CUDA technology, and the FFT operation was implemented via the CUFFT library. The software was developed in a Microsoft Visual C++ environment with the NI-IMAQ Win32 API (National Instruments). To achieve the high-speed data processing needed for the 3D video imaging at 10 Hz, we implemented graphics processing unit (GPU)-based real-time signal processing and visualization developed previously in our laboratory.^34,36

The signal processing flow chart of our system is shown in Fig. 2. Three threads were used to acquire raw data (Thread 1), implement GPU-accelerated FD-OCT data processing (Thread 2), and volume rendering (Thread 3). Thread 1 triggers Thread 2 for every B-scan and Thread 2 triggers Thread 3 for every complete C-scan, as indicated by dashed arrows. The solid arrows describe the main data stream; the hollow arrows indicate the internal data flow within the GPU. A C-scan buffer was placed in the host memory for the data transfer between GPUs because direct data transfer between GPU memories is currently not supported by CUDA. Using the GPU-based NUFFT algorithm, GPU-1 achieved a peak A-scan processing rate of $252,000\text{\hspace{0.17em}\hspace{0.17em}lines}/s$ and an effective rate of $186,000\text{\hspace{0.17em}\hspace{0.17em}lines}/s$ when the host-device data transferring bandwidth of PCIE-x16 interface was considered, which is higher than the camera’s acquisition line rate.

Fig. 2.

Signal processing flow chart of the dual-GPUs’ architecture. Dashed arrows, thread triggering; Solid arrows, main data stream; Hollow arrows, internal data flow of the GPU. Here the graphics memory refers to global memory.

The actual imaging speed of our OCT system was limited by the speed of the CMOS camera to $128,000\text{\hspace{0.17em}\hspace{0.17em}lines}/s$ for 1024-OCT mode or $70,000\text{\hspace{0.17em}\hspace{0.17em}lines}/s$ for 2048-OCT mode. Doubling the speed of the camera would double the speed of the OCT video image. OCT data sets are continuously acquired in real time, processed immediately, and visualized up to 12,800 A-scans/volume by either en face slice extraction or ray-casting-based volume rendering. The processing speed is fast enough to realize $10\text{\hspace{0.17em}\hspace{0.17em}volumes}/\mathrm{s}$ 3D FD-OCT “live” video image. To demonstrate $10\text{\hspace{0.17em}\hspace{0.17em}volumes}/\mathrm{s}$ 3D FD-OCT “live” video image, the acquisition line rate was set to be $128,000\text{\hspace{0.17em}\hspace{0.17em}lines}/s$ using the 1024-OCT mode. The acquisition volume size was then set to 12,800 A-scans and provided a $160(X)\times 80(Y)\times 1024(Z)$ voxel image after the signal processing stage. This process took less than 10 ms and leaves more than 90 ms for each volume interval at the volume rate of $10\text{\hspace{0.17em}\hspace{0.17em}volumes}/s$. The image plane is set to $512\times 512$ pixels, which means a total number of $512\times 512=262,144$ eye rays are used to construct the whole rendering volume for the ray-casting process. The actual rendering time is recorded during the imaging processing to be $\sim 3\mathrm{ms}$ for half volume and $\sim 6\mathrm{ms}$ for full volume, which is much shorter than the volume interval residual ($>90\mathrm{ms}$). The transverse resolution of the system was approximately 40 μm, assuming a Gaussian beam profile. We used this relatively low transverse resolution because each pixel in $X$ corresponds to $\sim 19$ microns; for the full-range complex OCT mode to work properly, the beam waist needs to be larger than the lateral separation between neighboring sampling points.³⁷ At this time the dynamic scenarios are all captured by free screen-recording software (BB FlashBack Express).

3. Real-Time OCT-Guided Microsurgical Procedures

The main hypothesis behind this work is that use of real-time 3D OCT video imaging to provide depth resolved views would enhance and extend surgical capabilities, improve surgical precision, and minimize human error in microsurgical settings. As a first step in assessing this hypothesis, we performed several imaging experiments to see whether our real-time OCT system as configured could provide real-time images that would be useful in guiding microsurgical maneuvers.

3.1. OCT-Guided Micro-Vascular Anastomosis

Vascular and microvascular anastomosis is considered to be the foundation of plastic and reconstructive surgery, transplant surgery, vascular surgery and cardiac surgery. In 1912 Alexis Carrel was awarded the Nobel Prize for describing a suture technique utilizing precise placement of sutures to connect the two ends of vessels thereby creating vascular anastomosis.³⁸ This procedure has remained a challenge for surgeons to master for over 100 years.³⁹ Even in the era of high-quality, binocular surgical microscopes equipped with optics providing highly magnified images; this technique still requires the highest level of skill and surgical expertise—especially for small vessels ($\text{diameter}<1.0\mathrm{mm}$).³¹

In the last two decades, innovative techniques have been introduced including vascular coupling devices,^40,41 thermo-reversible poloxamers,³⁹ and suture-less cuff techniques⁴² that can provide rapid vascular anastomosis, but there have been no notable innovations in the field of surgical imaging that provides an in-depth view and 3D guidance for microvascular surgery. Our pilot experiment with real-time 3D OCT video intra-operative guidance predict the emergence of a new field: Image-guided reconstructive microsurgery.

In our preliminary experiments we optimized the FD-OCT settings to visualize rat femoral arteries ($\text{diameter}<0.8\mathrm{mm}$). A video image sequence of one projection view with imaging size $5\mathrm{mm}\times 5\mathrm{mm}\times 3\mathrm{mm}$ $(250X\times 100Y\times 1024Z)$ and a frame rate of $5\text{\hspace{0.17em}\hspace{0.17em}volumes}/\mathrm{s}$ is shown in Fig. 3. In this mode we scanned a larger area by using a reduced frame rate to ensure that the complex conjugate part was removed. With ultrafast real-time intraoperative imaging, we were able to visualize the cut end of the vessels with all the layers of the vessel walls and lumen in six different views. Thus we could precisely place 11-0 sutures through the vessel wall without the use of an optical microscope. 3D image assistance was crucial in avoiding accidental suturing of the back wall of the vessel during anastomosis. In addition, we could obtain an optimized view of the conventional microsurgical instruments, needle, and thread in real time. Ultrafast imaging capture and display were sufficient to maneuver the instruments with advanced precision.

Fig. 3.

OCT-guided microvascular anatomosis: Rat femoral artery ($\text{diameter}<0.8\mathrm{mm}$) visualized through ultrafast 3D OCT imaging. (a) Cut end of the vessels visualized, (b) placement of sutures through proximal vessel wall, (c) and (d) needle driver and suture needle visualized traversing the vessel lumen, (e) suture thread connecting the two vessel ends visualized, (f) sutures tied to approximate the ends of the vessels. (V: vessel, SD: suture needle driver, SN: suture needle, ST: suture thread).

Based on our initial experience, FD-OCT-guided reconstructive microsurgery has a promising future; one that significantly advances the field of microsurgery and that may sow seeds for the emergence of a new field, that of “ultra microsurgery.” A major limitation of the present first generation approach is the narrow window of focus. This current shortcoming could be overcome by incorporating the OCT laser in a binocular optical microscope using a dichroic lens. 3D-OCT assistance during critical portions of microvascular anastomosis can provide better precision and can minimize human error. If the patient motion is small and the imaging depth is in the order of 3 to 5 mm, based on our limited experience, the OCT adjustment is not needed. However, patients can suddenly shift large distances and the imaging sites can go out of focus. To compensate for such an effect, we plan to use a foot control to move the position of the OCT scanning head up and down to bring the imaging site distance matched to the reference length. In addition, our OCT system can evaluate a vessel post anastomosis (e.g. diameter and patency) and can also quantitatively assess blood flow using speckle OCT³² as demonstrated by Jonathan et al.⁴³ These additional properties allow rapid detection of diminished flow or vessel narrowing at the anastomosis sites, thereby anticipating and avoiding potentially serious post-vascular anastomosis complications such as flap necrosis in plastic surgery.

3.2. OCT-Guided Retinal Microvascular Isolation

In our next study, we performed typical microsurgical tasks using the bovine eye as an ex vivo model. The retina is a very delicate and fragile neural tissue embryologically derived from brain. Extensive clinical and experimental literature describes the OCT anatomy of the retina in humans and in animals.^44–46 The tissue itself is optically transparent and has a distinct and clearly defined vascular supply. Isolating the microvasculature for the purpose of microvascular repair, microvascular anastomosis, or cannulation for drug delivery are all potential prototypical maneuvers that apply broadly across the field of microsurgery and may also become important retina-specific surgical maneuvers when consistently achievable on the micron scale.⁴⁷

In Fig. 4 we clearly show a direct correlation between the vascular anatomy in the gross figure and the OCT images acquired by our system. The imaging size is $3\mathrm{mm}\times 3\mathrm{mm}\times 3\mathrm{mm}$ $(\text{width}\times \text{length}\times \text{depth})$. The anatomical views correlate directly with the top view projection; additional information is provided by the front and back view projection images. Figure 5(a) to 5(e) is the selected top view projection images of the whole vascular isolation procedure, while Fig. 5(f) to 5(j) is the respective front view projection images. Figure 5(a) and 5(f) shows the imaging site before isolation. Microsurgical curved scissors are handled later by the surgeon to approach the undersurface of the upper vessel [Fig. 5(b) and 5(g)]. Then the upper vessel was lifted up gently using the scissors [Fig. 5(c) and 5(h)]. After this, the scissors are used as a blunt pick to elevate the retinal vessels from the neuronal substance [Fig. 5(d) and 5(i)]. The entire closed blades of the scissors support the retinal blood vessels, tenting them from the retinal surface and are seen in all projection images. Figure 5(e) and 5(j) shows the imaging site after the operation.

Fig. 4.

Comparison of vascular anatomy digital camera image and OCT image: (a) digital camera image, the scanning area is marked by red square; (b) front view projection of imaging site; (c) top view projection of imaging site; (d) back view projection of the imaging site.

Fig. 5.

A video image sequence showing the process of retinal microvascular isolation: (top view projection/front view projection) (a) and (f), before operation; (b) and (g), scissors approaching the bottom of the vessel; (c) and (h), scissors going through the top vessel; (d) and (i), scissor used as a blunt pick to elevate vessels from the neuronal substance; (e) and (j), vessel site after isolation. (V: vessel, S: scissors).

A similar maneuver is performed in Fig. 6 in which sutures are passed beneath the retinal vessels using a spatulated needle. Each of these vascular isolation techniques are broadly applicable across the field of microsurgery and may become increasingly useful in the field of retinal surgery as ultra-microsurgical techniques become increasingly refined and perfected.⁴⁸

Fig. 6.

Video image sequence showing the process of a suture passing through the vessel (V: vessel, SN: suture needle).

4. Conclusion

To provide real-time 3-D video imaging of the surgical site and tool tips while basic surgical maneuvers were performed, we used a novel microsurgical imaging system based on a full-range NUFFT Fourier-domain OCT (FD-OCT) integrated with a CPU-GPU heterogeneous computing architecture capable of OCT image processing and displaying at the effective maximum speed of $\sim 186,000$ A-scans/s and $10\text{\hspace{0.17em}\hspace{0.17em}volumes}/\mathrm{s}$.

By applying this ultrafast, real-time intraoperative imaging capability, we were able to visualize the cut end of the vessels with all the layers of the vessel walls and lumen in six different views. Thus we could precisely place 11-0 sutures through the vessel wall without the use of an optical microscope. 3D image assistance was crucial in avoiding accidental suturing of the back wall of the vessel during anastomosis. We could also obtain an optimized view of the conventional microsurgical instruments, needle, and thread in real time. The ultrafast imaging capture and display were sufficient to maneuver the instrument with advanced precision.

In addition, using ex vivo bovine eyes, application of this technology was extended to even smaller microvascular beds and potentially novel “ultra-microsurgical” applications in the neural retina. On the 0.1 mm scale of the retinal microvasculature we were able to repeat a prototypical surgical maneuver: single retinal vessel isolation with a handheld surgical micro-pick. The success of this maneuver was then confirmed by further isolation of the retinal vessels with a surgical suture.