In vivo virtual intraoperative surgical photoacoustic microscopy

Abstract

We developed a virtual intraoperative surgical photoacoustic microscopy system by combining with a commercial surgical microscope and photoacoustic microscope (PAM). By sharing the common optical path in the microscope and PAM system, we could acquire the PAM and microscope images simultaneously. Moreover, by employing a beam projector to back-project 2D PAM images onto the microscope view plane as augmented reality, the conventional microscopic and 2D cross-sectional PAM images are concurrently mapped on the plane via an ocular lens of the microscope in real-time. Further, we guided needle insertion into phantom ex vivo and mice skins in vivo.

A surgical microscope is a vital tool during operations since it started being used in otolaryngology.^{1, 2} In spite of the continuous improvement of the surgical microscopy performance, it could mainly provide the enlarged surface images without the sub-surface information. Because of this limitation, surgeons are required to have significant experiences and heavy trainings in current clinical practices. Thus, noninvasive visualization of the sub-surface information should play an important role during ophthalmic, micro-vascular, and neuro-surgeries.^{3, 4, 5}

Conventional optical microscopic techniques such as optical coherence tomography (OCT) and fluorescence microscopy (FM) has been applied for this purpose.^{6, 7, 8, 9, 10} OCT mainly provide microstructures of tissues based on optical scattering while the penetration depth of FM is significantly shallow and requires to use the exogenous contrast agents. However, real-time noninvasive mapping of microvasculatures are crucial to verify bleeding regions, guide vascular reconnection, or delineate angiogenesis during the surgeries.¹¹ To meet this purpose, we are convinced to develop a noninvasive surgical microscopy system to image microvasculatures and surgical intervention without any contrast agent beyond the skin surface in real-time.

Photoacoustic microscopy (PAM) is a rising imaging technique that combines optical excitation and ultrasound detection.^{12, 13, 14} Because PAM can provide label-free optical absorption information in a non-invasive manner with high contrast and resolution, this modality has widely been used to image tumor angiogenesis, tumor metabolism, brain function, ocular structures, molecular information, etc.^{15, 16, 17, 18, 19, 20}

In this letter, we developed a near-real-time virtual intraoperative surgical photoacoustic microscope (VISPAM) by integrating PAM and conventional surgical microscopy. The VISPAM system could obtain, process, and display both PAM and microscopic images at the same time. The Hilbert-transformed cross-sectional PAM B-scan images were projected back onto the microscope view plane as augmented reality through a home-made optical beam splitter, and then the PAM images were shown via the ocular lens mounted on the microscope. Thus, an additional tabletop display is unnecessary, so it is significantly convenient because surgeons potentially do not need to move their sights during the surgeries. We demonstrated the feasibilities of VISPAM in phantoms and live animals. Specially, via the in vivo the experiments, we delineated surrounding microvaculatures and guided needle insertion simultaneously in near-real time (i.e., two PAM B-scan images per second).

First, we implemented an optical-resolution PAM (OR-PAM) system. Fig. 1a shows the schematic of the VISPAM system. A Q-switched-diode-pumped-solid-state-laser (Elforlight, SPOT-10-200-532, 532 nm) with a repetition rate of 5 kHz was utilized as a laser source. The laser light was first attenuated by a neutral density filter (Thorlabs, NE02B) and divided by a beam splitter (Thorlabs, CM1-BP108). 8% of light was detected by a photodiode (Thorlabs, SM05PD5A) to synchronize galvo scanners and a data acquisition (DAQ) system. The rest of the light was used for PAM. Then, we developed a VISPAM probe by adapting with a commercial surgical microscope (Carl Zeiss, OPMI). The VISPAM probe consisted of three main devices: (1) a display projector, (2) beam splitting, (3) PAM scanning subsystems. Each subsystem was notified as numbers “1,” “2,” and “3,” respectively. Fig. 1b shows the corresponding photograph.

Figure 1.

(a) Schematic of the VISPAM system. (b) Photograph of the VISPAM probe. (c) Axial (left) and lateral (right) profiles of a carbon fiber (6 μm). (d) VISPAM B-scan image of a needle inserted obliquely in a living mouse leg tissue. COM, computer; PD, photo diode; BS, beam splitter; NF, neutral density filter; AMP, amplifier; BP, beam projector; M, mirror; G, galvo scanner; OL, object lens; UT, ultrasonic transducer; WT, water tank; and S, sample.

The display projector subsystem 1 composed of a beam projector (Optoma, PR320) and two mirrors to control the beam path. The subsystem 2 (i.e., the beam splitting device) played a role to project the acquired PAM images back onto the field of view of the conventional microscope through the microscopic ocular lens. A custom-made mount which was attached to the commercial microscope held the beam splitter. The PAM scanning subsystem 3 composed of two galvo scanners (Thorlabs, GVS001), an objective lens (Thorlabs, AC254-060-A), and a beam splitter (Edmund optics, NT68-395). This beam splitter (40% of reflection and 60% of transmission) was designed to reflect the 532 nm beam (i.e., PAM light) and transmit the visible light (i.e., light for the optical microscope). Thus, 40% of the reflected 532 nm beam irradiated into the sample. Further, 60% of the reflected visible light transmitted via the beam splitter and the ocular lens. The two galvo scanners were used for optical scanning. The generated PA signals from the samples were obtained by an unfocused ultrasonic transducer (Olympus NDT, V312, and a center frequency of 10 MHz). We used a small water tray for ultrasound coupling. The acquired PA signals were amplified by a pulser/receiver (Olympus NDT, 5072PR) and then digitized by a DAQ board. Typically, 200 of PA A-line scans were utilized to form one PA B-scan image, and the image was displayed at 2 frames per second. The B-scan image was back-projected via the display projector subsystem into the microscopic image plane via the ocular lens. The axial and lateral resolutions were measured by imaging a carbon fiber (6 μm) in water, and they are 131 and 17 μm, respectively, as shown in Fig. 1c. Fig. 1d indicated the maximum tissue penetration (∼900 μm) in a living mouse thigh, experimentally measured by inserting a needle obliquely.

To demonstrate the performance of VISPAM, we guided a needle into four black hairs contained in a gelatin phantom. The field of view (FOV) of the back-projected PA B-scan was 6 mm × 5 mm along the x and z axes, respectively. As shown in media 1, we have guided the needle into the phantom containing the hairs. Figs. 2a, 2c, and 2e show the screen-shots acquired via the left ocular lens during the needle guidance. Figs. 2b, 2d are the close-ups of the PAM B-scan images overlaid in Figs. 2a, 2c, respectively. The cross-sections of the hairs and needle are clearly visible in the PAM images. A PAM maximum amplitude projection (MAP) image is shown in Fig. 2f. The FOV of the PAM MAP image is 6 mm × 6 mm along the x and y axes, respectively, and it took approximately 2 min to acquire the one MAP image. The needle and hairs are clearly mapped in the MAP image.

Figure 2.

Download video file ^{(1.7MB, avi)}

Real-time VISPAM in phantoms. (a), (c) The screen-shots of overlapped PAM and surgical microscopic images obtained via the left ocular lens during needle intervention to four black hairs embedded in a gelatin phantom. (b), (d) The enlarged PAM B-scan images extracted from (a) and (c), respectively. (e), (f) The screen-shot and PAM MAP image of the hairs and needle in the phantom, respectively. H, hair and N, needle (enhanced online).

To investigate the feasibility of VISPAM in vivo, we guided the needle insertion into the mouse skin. All animal experimental procedures satisfied the laboratory animal protocol approved by the institutional animal care and use committee. A healthy BALB/c mouse that weighted ∼20 g was used in animal experiments. The mouse was anesthetized with intraperitoneal injection of a mixture of xylazine (15 mg/kg) and ketamine (85 mg/kg body weight). After removing the hairs on the left thigh, the mouse was located on a homemade animal holder. As shown in media 2, we inserted the needle into the mouse thigh while we monitored the procedures using the VISPAM probe. The photograph of the mouse and needle insertion is shown in Fig. 3e. The laser pulse energy on the mouse skin was less than 17 mJ/cm², which was below the American National Standards Institute (ANSI) safety limit (20 mJ/cm²). Figs. 3a, 3c show the screen-shots captured via the left ocular lens during the needle intervention. Figs. 3b, 3d are the magnified PAM B-scan images extracted from Figs. 3a, 3c, respectively. The cross-sections of the blood vessels and needle are clearly visible in the images. The needle and surrounding vasculatures are clearly visualized in the PAM MAP image, which was averaged 5 times to enhance the signal to noise ratio as shown in Fig. 3f.

Figure 3.

Download video file ^{(1.8MB, avi)}

Real-time VISPAM in vivo. (a), (c) The screen-shoots, acquired via left ocular lens, of overlaid the PAM and surgical microscopy images presenting microvasculatures in the left thigh of the mouse and needle insertion. (b), (d) The magnified PAM B-scan images cut from (a) and (c), respectively. (e) The photographs of the mouse (inset) and mouse skin with needle insertion. (f) The PAM MAP image showing the needle inserted into the skin and surrounding blood vessels. V, vessel and N, needle (enhanced online).

In summary, we developed a VISPAM system by fusing PAM and conventional microscopy. Unlike previously developed advanced surgical microscopy, VISPAM can potentially guide surgical processes and visualize surrounding microvasculatures simultaneously in real time without any injection of contrast agents. By employing augmented reality, we could project the PAM B-scan images on the microscopy image view plane. Thus, no extra PAM display is required, which can significantly enhance clinical translation. In the future, three issues needs to be addressed: (1) noncontact detection of photoacoustic signals to remove water or ultrasound gel coupling,²¹ (2) improvement of the image display rate using graphics processing unit,²² and (3) the use of an invisible near-infrared laser source to minimize the surgical disturbance from the visible PA laser light.