Development of a real-time spectral imaging system using in-site micro-LED-based illumination and high-speed micro-camera for endoscopic applications

Abstract

We designed a compact, real-time LED-based endoscopic imaging system for the detection of various diseases including cancer. In gastrointestinal applications, conventional endoscopy cannot reliably differentiate tumor from normal tissue. Current hyperspectral imaging systems are too slow to be used for real-time endoscopic applications. We are investigating real-time spectral imaging for different tissue types. Our objective is to develop a catheter for real-time hyperspectral gastrointestinal endoscopy. The endoscope uses multiple wavelengths within UV, visible, and IR light spectra generated by a micro-LED array. We capture images with a monochrome micro camera, which is cost-effective and smaller than the current hyperspectral imagers. A wireless transceiver sends the captured images to a workstation for further processing, such as tumor detection. The spatial resolution of the system is defined by camera resolution and the distance to the object, while the number of LEDs in the multi-wavelength light source determines the spectral resolution. To investigate the properties and the limitations of our high-speed spectral imaging approach, we designed a prototype system. We conducted two experiments to measure the optimal forward voltages and lighting duration of the LEDs. These factors affect the maximum feasible imaging rate and resolution. The lighting duration of each LED can be shorter than 10 ms while producing an image with a high signal-to-noise ratio and no illumination interference. These results support the idea of using a high-speed camera and an LED-array for real-time hyperspectral endoscopic imaging.

1. INTRODUCTION

In 2020 in the United States, the digestive system cancers are ranked the first with more than 333,000 and 167,000 incidences and mortality, respectively [1]. As one of the standard diagnostic methods, gastrointestinal (GI) endoscopy plays a critical role in the diagnosis of GI cancers. GI cancer screening by endoscopy illustrated a strong impact on localized cancer diagnosis rates [2]. However, conventional GI endoscopy still has its limitations. For example, about 8 to 11% of tumors are missed due to the lack of visibility during upper GI endoscopy [3, 4]. One potential solution to improve the diagnostic accuracy of endoscopy is the use of hyperspectral/multispectral imaging (HSI/MSI). It facilitates the diagnosis of abnormalities which are not visible through conventional RGB imaging [5].

There are some limitations with previously developed multi- or hyper-spectral endoscopes presented in the literature. Some limitations include low imaging rate (non-real-time), low spatial or spectral resolution, and bulky systems. Some of these issues were previously investigated [6-10]. However, there are still limitations that need to be addressed. In some of these endoscopes [6, 8, 9], fiber optic bundles were used to deliver the light to the imaging site or transmit images to an external camera. This approach makes the illumination and imaging flexible but limits the physical flexibility of the endoscope as fiber optic cables have a limited bend radius. In some other designs [7-10], the low imaging rate and low spectral or spatial resolutions are still valid challenges.

In our design, we use an LED array as an in-situ spectral light source to address the need for fiber optics for light delivery. It increases the mobility of the endoscope. Furthermore, we use a wireless connection between the endoscope and an external workstation for delivering the data from and sending the commands to the endoscope. It helps to remove the data wires and improve the mobility of the endoscope. Our goal is to design and implement a real-time spectral imaging endoscope. The wavelengths of the LEDs range from ultraviolet (UV) and visible light spectrum (~350 nm to ~700 nm) to infrared (~700 nm to ~1250 nm). The main objective of this study is to evaluate the feasibility of using an LED-array light source with a monochrome camera for real-time wavelength scanning spectral imaging.

2. MATERIALS AND METHODS

2.1. Hyperspectral endoscopic system architecture

The proposed endoscopic system should image the GI track in real-time. For an HSI system, the frame rate should be higher than 20 to be considered real-time [8]. In addition, the spatial and spectral resolutions are the main factors that define the quality of a hyperspectral endoscope. Our goal is to design and implement a real-time endoscope with a minimum spatial resolution of 0.2 mm and a spectral resolution of up to 30 discrete wavelengths.

In an HSI system, the maximum frame rate of the camera and the number of scanned wavelengths are two factors that limit the hyperspectral frame rate. In this study, we use a multi-wavelength LED array with up to 30 wavelengths as our light source. Thus, the maximum spectral resolution of our system could be 30. Another important specification of the system is the size. Here, we use micro LEDs with a maximum footprint of 0.5 mm × 0.5 mm. The LEDs are mounted on a PCB surrounding the camera.

Figure 1 shows the block diagram of the proposed endoscopic system. The proposed imaging system consists of two parts; an internal catheter for image acquisition and an external workstation for further processing. The image acquisition part at the tip of the catheter is composed of imaging, processing and controlling, and transceiver modules. The imaging module consists of a camera and a multi-wavelength LED array. Figure 2 shows the schematic diagram of the catheter. The workstation is a computer that is responsible for creating controlling commands for the image acquisition and processing of the captured image data. The processing consists of image denoising, image channels integration, and image classification.

Figure 1.

The schematic block diagram of the hyperspectral endoscopic system.

Fig. 2.

The schematic diagram of the proposed hyperspectral endoscope catheter.

2.2. Illumination and Imaging characteristics

The quality of endoscopic imaging is directly affected by the performance of the imaging module. There are many different factors that play roles in the imaging system quality and speed such as camera imaging rate and quality, LED array illumination characteristics, and controlling process speed. We conducted two experiments to test and evaluate the basic capabilities of the proposed imaging system. For these experiments, we designed and printed a simple spectral evaluation template (as shown in Figure 3) to evaluate the quality of the system in seven different wavelengths including 405 nm, 505 nm, 555 nm, 610 nm, 660 nm, 700 nm, and 740 nm. The size of the evaluation template was 50 mm × 50 mm and had seven 7 mm × 30 mm bars, each corresponding to one wavelength, as well as black and white sections and focusing crosshair markers. The template has been coated with an anti-reflective material for glare reduction during imaging. As seen in Figure 3, the colors in the printed template might be slightly different from the desired colors on the designed template due to printing limitations. This color shift could be a source of error during our evaluation experiments. We implemented the template with a monochrome micro digital camera and an LED array inside a dark box to block the environment light during the experiments. Figure 4 shows the setup we used for the experiments. We conducted these experiments to evaluate the characteristics of the LED array and its impact on imaging parameters such as imaging rate and noise level.

Figure 3.

The designed hyperspectral evaluation template (A) and the printed template (B).

Figure 4.

(A) Experimental setup for the feasibility study. (B) LED array, camera, and spectral template inside the dark box.

2.3. Evaluation metrics

We measured noise level and light reflection rate as the imaging quality metrics in this study. The noise level was assessed using signal-to-noise ratio (SNR). We measured SNR in decibel (SNR_dB) using Equation (1) where μ and σ are the average reflectance (image intensity) and standard deviation of the intensities at the corresponding band, respectively.

$$SN{R}_{dB}=20lo{g}_{10}\left(\frac{\mu }{\sigma }\right).$$ (1)

The image reflection rate was the ratio of the image intensity to the full reflection (i.e., the intensity level of 255 in an eight-bit dynamic range).

2.4. Experiments

Experiment I: We conducted the first experiment to measure the impact of lighting duration of LEDs controlled by forward voltage pulse width (PW) shown in Figure 5 on the imaging quality of the spectral imaging system. In this experiment, we forward biased an LED using pulse waves with different PWs and recorded a snapshot per pulse wave. Figure 5 shows the driving signal applied to the LEDs. For generating and applying the pulses with different PWs we used the FPGA. For each image, we calculated the SNR at the corresponding region of interest (ROI) on the image. The ROI is the template bar that corresponds to the illumination wavelength. We repeated this experiment twice using two LEDs with wavelengths of 505 nm and 660 nm.

Figure 5.

Forward bias pulse of LEDs.

Experiment II: In this experiment, we measured light reflection rate and the imaging SNR in different forward voltage levels. We turned on each LED using different forward voltage levels. We put a potentiometer in series with LEDs to control its illumination by changing its forward voltage. Then, for each adjusted voltage, we captured an image and calculated the SNR_dB and the percentage of the reflected light from the corresponding ROI. We repeated this experiment using LEDs with wavelengths of 405 nm, 505 nm, 555 nm, and 660 nm. Figure 9A shows the SNR of the captured images based on the forward voltage applied to the LED. The results of this experiment could be used for optimizing the LEDs’ forward voltages to have a reasonable imaging quality with consistent reflection across all the wavelengths while the power consumption is minimized.

Figure 9.

Seven-channel data cube. The yellow arrows show the corresponding ROIs for the channels.

According to the specifications of the LEDs, the delay time to turn the LEDs either on or off are in the order of tens of nanoseconds which is short enough for our applications to have LED switching with no illumination interference between the LEDs. After completing the pulse width and forward voltage optimizations, we performed multispectral imaging using the seven-LED light source. The maximum frame rate of the camera used in this study was 120 fps.

3. RESULTS

Figure 5 shows the captured images when different PW was applied to the LEDs. The top row images were captured when the pulses were applied to the LED with a wavelength of 505 nm and the bottom row images were captured when the pulses were applied to the LED with a wavelength of 660 nm. To generate low noise images, we also applied a DC forward voltage to each LED and captured several frames. By using an averaging filter across the captured frames, a low noise image per wavelength is generated as a reference image. In Figure 6, the first images from the left on each row show the low noise images for the corresponding LEDs. The graph in Figure 7 shows the SNR_dB versus the pulse width for the two tested LEDs.

Figure 6.

Captured images for different pulse widths (top: LED of 505 nm, bottom: LED of 660 nm).

Figure 7.

SNR_dB versus the pulse width applied to the LED forward voltage.

Figure 8 shows the impact of LED forward voltages on SNR and light reflection rate measured for four different LEDs during Experiment II. We optimized the forward voltages for all seven LEDs based on these results and set the lighting duration time and the delay between illuminations both to 10 ms. Figure 9 shows a sample datacube using the optimum setting. The seven-channel spectral imaging rate was about 7 hypercubes per second using the proposed approach with a monochrome camera with a 120 fps imaging rate. The SNR_dB of the datacube was 18.9 dB.

Figure 8.

(A) SNR_dB versus LED forward voltage. (B) Light reflection versus forward voltage for four sample LEDs

4. CONCLUSIONS

We proposed an approach for a real-time micro-LED-based spectral imaging system for endoscopic applications. We conducted a feasibility study to characterize the in-site LED-based illumination for wavelength scanning spectral imaging. For a hyperspectral endoscope, some of the main imaging factors are the imaging frame rate and spatial and spectral resolutions. The results of the experiments in this study show that using a micro-LED array combined with a monochrome camera could be used for a high-speed wavelength scanning hyperspectral imaging with relatively high spatial and spectral resolutions. The preliminary results showed that an in-situ LED-based light source could be fast enough for real-time or near-real-time, high quality (low-noise) imaging. The imaging module of the real-size spectral endoscope system will be implemented and packaged in a small enough scale to be accommodated at the tip of a real-size endoscopic catheter. We will use micro-CMOS LEDs with footprint sizes smaller than 0.5 mm × 0.5 mm and design a donut shape PCB for the LED array with a diameter of less than 12 mm. In addition, we will use a high-speed monochrome camera with a frame rate of around 600 fps to decrease the acquisition time of the spectral endoscope system. We will use FPGA prototyping for the controlling and processing core and will use that prototype to design and fabricate an ASIC as a replacement for the FPGA. Using an ASIC helps to decrease the size, power consumption, and cost of the system. The real-time micro-LED based hyperspectral endoscopic imaging technique can have a variety of applications for disease detection of gastrointestinal and other organs.