Multiresolution foveated laparoscope with high resolvability

Abstract

A key limitation of the state-of-the-art laparoscopes for minimally invasive surgery is the tradeoff between the field of view and spatial resolution in a single-view camera system. As such, surgical procedures are usually performed at a zoomed-in view, which limits the surgeon’s ability to see much outside the immediate focus of interest and causes a situational awareness challenge. We proposed a multiresolution foveated laparoscope (MRFL) aiming to address this limitation. The MRFL is able to simultaneously capture wide-angle overview and high-resolution images in real time; it can scan and engage the high-resolution images to any subregion of the entire surgical field in analogy to the fovea of human eye. The MRFL is able to render equivalently 10 million pixel resolution with a low data bandwidth requirement. The system has a large working distance (WD) from 80 to 180 mm. The spatial resolvability is about 45 μm in the object space at an 80 mm WD, while the resolvability of a conventional laparoscope is about 250 μm at a typically 50 mm surgical distance.

Laparoscopes have been widely adopted for many clinical applications, especially in minimally invasive surgery (MIS). The state-of-the-art laparoscopic technologies, however, have several limitations [1]. First of all, the major limitation is a tradeoff between the spatial resolution and the field of view (FOV) [2]. With the state-of-the-art laparoscopes, to see fine details of a narrow surgical field, surgical procedures are usually performed at a zoomed-in view, where the laparoscope is used at a short working distance (WD), typically less than 50 mm. A highly zoomed-in view leads to the loss of peripheral vision and awareness of situations occurring outside the immediate focus area of the laparoscope, which may cause fatal problems in some extreme cases such as unawareness of excessive bleeding. This limitation is clinically addressed by manually moving the entire laparoscope in and out of the camera port to obtain either close-up views or wide-angle overviews. A trained assistant is required for maneuvering the camera.

Second, the practice of frequently maneuvering the camera by a trained assistant introduces ergonomic conflicts between the surgeon and the assistant. Robotically assisted systems have been developed to eliminate the need for a human camera holder. However, delays in task performance are reported due to errors in voice recognition or robotic control of camera speed, and also significant practice is required to become efficient with setup and use [3]. The ergonomic conflict is aggravated with the increasingly popular single port access (SPA) technique. Port-grouping in SPA procedures raises a number of problems. One of the most significant aspects is the limited spacing. It has been suggested that varying the magnification of the laparoscope and making it a low profile can reduce the effect of crowding [4].

Several technologies have been developed and applied to laparoscopes to alleviate the FOV-resolution tradeoff. For instance, digital zoom is a common technique adopted for effectively changing the FOV coverage, but it does not improve the optical resolution. The foveated lens design technique [5] extends the FOV by introducing a large amount of distortion to the peripheral field, which leads to much lower spatial resolvability of the peripheral field than that of the central field. The fluidic lens laparoscopic zoom camera [6] has a small volume and a long WD and can vary its FOV. It may effectively reduce the effect of ergonomic conflicts, but it is not able to simultaneously capture wide-angle and zoomed-in views. The dual-view endoscope system [7] offers the ability to capture both wide-angle and zoomed-in views simultaneously, but the system is limited by the low resolution and low light throughput.

We proposed a multiresolution foveated laparoscope (MRFL), which has the potential to make MIS procedures more efficient and safer. The conceptual system layout is shown in Fig. 1. The MRFL is able to simultaneously acquire both wide-angle overview and high-resolution zoomed-in images of a surgical area integrated through shared objective and relay lenses. A dual-view MRFL prototype was designed and implemented, of which the wide-angle probe captures an 80° FOV for situational awareness and the high-resolution probe covers a 26° FOV of interest with as high as 45 μm resolution in the object space for accurate procedures. The high-resolution probe can be engaged at any subfield of the entire surgical field in real time by using a 2D scanner. The MRFL prototype system equivalently yields a resolution of more than 10 million pixels with two HD image sensors. The optical system was designed so that a standard 10 mm diameter package was adapted for housing the objective and relay groups along with illumination fibers, which facilitates the rapid adoption of the developed technology for clinical use. Additionally, the prototype system was optimized for a long WD from 80 to 180 mm. It was further designed to achieve flexible length profiles, a normal profile for a standard multiport procedure, and a low profile for SPA procedures, by allowing a reconfigurable number of relay groups. Benefiting from the long WD, a low-profile MRFL system can be mounted at a fixed pose through a camera port and can potentially reduce the conflicts with other surgical instruments and reduce the need for additional incisions and thus cut down on scarring and healing time.

Fig. 1.

Conceptual system layout.

The optical layout of the MRFL system is shown in Fig. 2. It consists of a shared f/2.5 objective lens group, multiple shared relay lens groups, a scanning lens group, a wide-angle imaging probe, and a high-resolution foveated imaging probe. To fit the standard 10 mm diameter package, the diameters of the objective lens and the rod lens relay groups need to be less than 5.7 mm. The objective lens captures the entire 80° FOV with a diffraction-limited performance. It was designed to be telecentric in image space. The relay lens groups were designed to be telecentric in both object and image space, and each relay group works at a magnification of −1. The telecentricity of the objective and relay groups enables a flexible number of relay lens groups of limited diameter to be concatenated, without noticeable degradation of image quality, to create laparoscopes of different length profiles. To reduce fabrication cost, each of the relay groups was designed to be identical, and the left and right portions within each relay group are symmetric.

Fig. 2.

System layout of MRFL: (a) objective lens group, (b) one relay lens group, and (c) scanning lens group.

The scanning lens group works like an eyepiece with a 2D scanning mirror placed at the exit pupil. A polarizing beam splitter (PBS) along with a quarter-wave plate is inserted between the scanning lens and the mirror for splitting the light paths for the wide-angle and high-resolution probes. The PBS directs the s-polarized light into the wide-angle imaging probe and p-polarized light into the high-resolution imaging probe. Although a PBS and a quarter-wave plate were used to improve the light efficiency, compared to a standard single-view laparoscope, only half of the collected light goes into the high-resolution probe, and the other half goes into the wide-angle probe. The anticipated light-splitting effect was compensated for by the fact that the F/2.5 objective lens has twice as much light collection capability as a standard F/4 objective lens. Moreover, the lower F-number objective lens provides better resolution.

The optical system design of the MRFL system was quite challenging due to the limited lens diameter, large FOV, telecentric requirements, and low F number. The objective lens was designed to have a focal length, f_obj, of 2 mm and have diffraction-limited performance with 17% distortion. The length of each relay group is 80 mm. Each group has a symmetric configuration to take advantage of the fact that the odd-order aberrations such as distortion, coma, and lateral chromatic aberration are canceled out. Each group was well optimized with diffraction-limited performance so that the image quality will not degrade much when the number of relay groups increases.

The scanning lens group requires a long exit pupil distance (EPD) to accommodate for the PBS and wave plate. The focal length of the scanning lens, f_scan, is 14 mm, while the EPD is 16 mm. However, one problem is that as the number of relay groups increases, the spherical aberration and the longitudinal chromatic aberrations accumulate. In order to correct those accumulated aberrations through multiple relay groups, we apply a diamond-turned plastic hybrid lens in the scanning lens group. One surface of the plastic lens has a diffractive optical element (DOE), and the other surface is aspheric. The DOE has the opposite dispersion compared to the refractive lens, so that the longitudinal chromatic aberrations can be balanced. The aspheric surface corrects the higher-order aberrations. The plastic lens was placed near the intermediate pupil to effectively balance those accumulated aberrations.

Both the wide-angle and high-resolution imaging probes use simple optics. The wide-angle probe consists of a doublet and a field lens, and the high-resolution probe has two singlets and a field lens. The focal length of the wide-angle imaging probe is 30 mm, while that of the high-resolution imaging probe, f_high-res, is 90 mm.

By controlling the tilting angle of the scanning mirror, the high-resolution probe can be engaged to any subfield of the entire surgical area. Assuming the high-resolution probe is aiming at θ° in the entire FOV, the tilting angle of the scanning mirror, β°, is given by

$$\beta =0.5\times (f_{\text{obj}}/f_{\text{scan}})\times \theta .$$ (1)

In our prototype design, the maximum tilting angle required is 1.9° to cover the full FOV. A motorized gimbal mirror mount (Zaber T-OMG Series) was used, which has a tilting range of ±7°, a maximum speed of 7°/s, and a minimal scanning step of 0.0001°. The mirror enables the ability to scan across the entire surgical field in less than 0.4 s and to fix the position of the high-resolution probe with an accuracy of 0.097 mm.

The MRFL was optimized at the WD of 120 mm with a depth of field from 80 to 180 mm. The corresponding surgical field is about 80 mm × 60 mm at an 80 mm WD and 240 mm × 180 mm at 180 mm distance, respectively. For the high-resolution probe, of which the FOV is 1/3 of that of the wide-angle probe, the visual surgical field is 27 mm × 20 mm at an 80 mm WD and 80 mm × 60 mm at a 180 mm WD.

In the prototype we used 1/3″ color CCD sensors (PointGrey DragonFly2 DR2-13S2C-CS) for both imaging probes. The pixel resolution of the sensors is 1280 × 960, and the color pixel size is 7.5 μm × 7.5 μm. The spatial resolvability of the high-resolution probe is given by

$$D=7.5um/(m_{\text{high-res}}\times m_{\text{obj}}),$$ (2)

where m_obj and m_high-res are the magnifications of the objective lens and the high-resolution imaging probe, respectively. At a giving WD, L, they are calculated by m_obj = −f_obj/L and m_high-res/f_scan, respectively. The spatial resolvability of the high-resolution probe is 46.67, 70, and 105 μm, corresponding to a spatial frequency of 10.7, 7.1, and 4.8 lps/mm, at a WD of 80, 120, and 180 mm, respectively.

The MTF curves of the wide-angle and the high-resolution imaging probes at 120 mm WD are shown in Fig 3. For both imaging probes, the image contrast is larger than 0.15 at 66 lp/mm, which corresponds to the cutoff frequency of the image sensors in the image space. The high-resolution probe has the same diffraction-limited performance when β = 1.9°.

Fig. 3.

(a) MTF of high-resolution probe when β = 0° and (b) MTF of wide-angle probe.

Figure 4(a) shows the system prototype. The normal profile package has four relay groups, and the low-profile package has two relay groups. Figure 4(b) shows an image captured by the wide-angle probe with an abdominal cavity model placed at a 120 mm WD. The size of the model is 160 mm × 120 mm. Figure 5 shows a preliminary test result captured by the high-resolution probe with a US1951 resolution target placed at an 80 mm WD. Group 3 element 3 can be resolved, which corresponds to 10.10 lps/mm in object space (bar width 49.5 μm). A display interface is designed so that the high-resolution image can be embedded into the wide-angle image, in analog to the fovea of the human eye.

Fig. 4.

(a) MRFL prototype and (b) wide-angle image of an abdominal cavity 120 mm away from the MRFL.

Fig. 5.

High-resolution image at 80 mm WD.

The MRFL system has superb optical quality and field coverage compared with a conventional laparoscope. Assuming a typical 70° FOV and 50 mm WD, the corresponding visible area of a conventional laparoscope is only 56 mm × 42 mm with a spatial resolvability of about 2 lps/mm in the object space. With the MRFL at an 80 mm WD, the surgical area by the wide-angle probe is about two times that of the conventional one. More prominently, the resolvability of the high-resolution probe is more than 10 lps/mm in the object space, which is five times that of the conventional one. With the MRFL at an 180 mm WD, the area covered by the wide-angle probe is more than 18 times that of the conventional laparoscope, while the area covered by the high-resolution scope is twice that of the conventional one. The resolvability of the high-resolution probe in the object space is 4 lps/mm, which is twice as good as that of the conventional one.

In conclusion, we developed an MRFL that can provide both wide-angle and high-resolution images of a surgical area. With this foveated and multiresolution capability, this device is anticipated to provide surgeons good situational awareness and better resolution during laparoscopic surgeries; thus it can help provide better and safer surgeries for patients. The device is able to equivalently render 10 million pixel resolution (9 × 1280 × 960) by using only two HD (1280 × 960) cameras. Compared to the use of a superresolution camera, the MRFL system can save much bandwidth and have a faster frame rate, which is required in laparoscopic surgery. Moreover, the MRFL can reduce the space limitation in SPA procedures. The system will be tested under animal models.