Processing-Aware Real-Time Rendering for Optimized Tissue Visualization in Intraoperative 4D OCT

Abstract

Intraoperative Optical Coherence Tomography (iOCT) has advanced in recent years to provide real-time high resolution volumetric imaging for ophthalmic surgery. It enables real-time 3D feedback during precise surgical maneuvers. Intraoperative 4D OCT generally exhibits lower signal-to-noise ratio compared to diagnostic OCT and visualization is complicated by instrument shadows occluding retinal tissue. Additional constraints of processing data rates upwards of 6GB/s create unique challenges for advanced visualization of 4D OCT. Prior approaches for real-time 4D iOCT rendering have been limited to applying simple denoising filters and colorization to improve visualization.

We present a novel real-time rendering pipeline that provides enhanced intraoperative visualization and is specifically designed for the high data rates of 4D iOCT. We decompose the volume into a static part consisting of the retinal tissue and a dynamic part including the instrument. Aligning the static parts over time allows temporal compounding of these structures for improved image quality. We employ a translational motion model and use axial projection images to reduce the dimensionality of the alignment. A model-based instrument segmentation on the projections discriminates static from dynamic parts and is used to exclude instruments from the compounding. Our real-time rendering method combines the compounded static information with the latest iOCT data to provide a visualization which compensates instrument shadows and improves instrument visibility.

We evaluate the individual parts of our pipeline on pre-recorded OCT volumes and demonstrate the effectiveness of our method on a recorded volume sequence with a moving retinal forceps.

1. Introduction

Intraoperative Optical Coherence Tomography (iOCT) in ophthalmology has seen dramatic technical advances in imaging speed and quality in recent years. Technological advances such as the application of swept-source lasers [8], spectral splitting [7] and linear velocity spiral scanning [3] have made continuous 4D intraoperative OCT imaging feasible at high resolution and volume rates. State of the art systems are able to sample with volume rates of up to 24.2Hz at a resolution of 330×330×595 voxels [9]. Ehlers et al. [4] have shown the effectiveness of 2D iOCT in clinical practice in a large scale study, however clinical studies on 4D OCT do not yet exist. Still, 4D OCT is expected to improve spatial visualization during complex and precise maneuvers. Potential applications include robotics [18] or even retinal surgery under exclusive OCT guidance [9] which can greatly reduce adverse effects associated with the endoilluminators required for traditional fundus view surgery. With data rates surpassing 6GB/s, 4D iOCT is a uniquely challenging modality for advanced processing in an intraoperative setting. OCT-only surgery is currently challenged also by higher noise level compared to diagnostic OCT, instrument shadows hiding relevant anatomical structures and a limited field of view.

In this work, we propose a real-time processing and rendering pipeline to improve instrument visibility by combining volumes temporally. A semantic segmentation step allows to discern static retinal tissue, which is then used to register subsequent volumes and selectively mitigate instrument shadows. We propose an adaptive visualization that makes the involved processing steps apparent to the viewer.

To improve quality in diagnostic OCT, repeated scanning of B-scans is employed to reduce noise by averaging[15,1]. Because this approach would reduce volume rates to impractical speeds, a similar solution is to average subsequent volumes, however this requires some form of motion compensation. Previous works on OCT motion compensation are focused on motion during acquisition of a single volume [10] or on registering diagnostic OCT volumes acquired at different visits for progression analysis. 3D SIFT features [12] have been used for full 3D registration but many approaches use 2D projections generated from volumes to perform separate alignment across dimensions using ICP [13], SIFT [6] or SURF [14]. To the best of our knowledge, there is no published research on solving the problem with the additional constraints of real-time processing and a potentially moving surgical instrument.

Metallic instruments pose an additional challenge to iOCT visualization, causing total loss of signal below the instrument due to the metal blocking the light. This shadows relevant retinal tissue below the instrument, causing ”holes” in the 3D rendering. Special OCT-compatible instruments have been proposed [5], but are not yet translated to clinical practice. Effective visualization of iOCT volumes is not trivial due to the aforementioned problems, which is why straightforward methods like direct volume rendering (DVR) with intensity-based transfer functions perform badly (c.f. Fig. 1, left). Viehland et al. [16] introduced a DVR method for OCT that was extended by Draelos et al. [2] to add colorization along the axial direction. Both however rely on spatial filtering with Gaussian and median filters to reduce noise prior to rendering. Recently, Weiss et al. [17] introduced a layer-aware volume rendering in which a layer segmentation was used to anchor a color map to improve perception (c.f. Fig. 1, middle). In this work, our aim is to show the feasibility of advanced real-time processing and visualization for 4D iOCT. Our novel real-time processing and rendering method combines temporal information and mitigates instrument opacity (c.f. Fig. 1, right).

Fig. 1.

Challenges and our solution to 4D iOCT rendering: Straightforward DVR (left) is not sufficient for good visualization and advanced color schemes[17] (middle) do not cope well with noise and artifacts. Our novel method (right) reduces noise and artifacts and extends the field of view. All images were rendered with the same opacity transfer function. See supplementary material for a high-resolution version.

Our contributions are the following: We introduce a fast registration and compositing method that temporally combines the static parts within the OCT view. This is supported by a learning-based instrument segmentation in 2D projection images. We then propose a processing-aware rendering method that optimizes tissue visibility by inpainting shadowed areas, ensuring interpretability of the view through sensible color mapping.

2. Methods

Our visualization was built around the two goals of reducing imaging noise over time and using data from previous frames to fill in the gaps produced by instrument shadowing. The approach is based on the notion that with 4D iOCT, retinal tissue is imaged many times while remaining relatively unchanged. Therefore, we discriminate the retinal tissue to align it over time and use the aggregated information to improve our visualization. Due to the high data rates and time constraints we rely on processing 2D projection images instead of 3D volumes.

Our novel method consists of several steps (see Fig. 2): To initialize our algorithm, a larger reference volume is acquired to prime our compounding volume I_c and establish a reference. For each newly acquired volume I_a and once for the reference volume, we generate a set of 2D projection images and use a learning-based approach to find a tissue mask M_t by segmenting instruments in 2D. The masked projection images are then used to register the I_a to I_c. The registration together with M_t allows us to maintain an updated representation of the static retinal tissue while excluding the instrument and shadows. Finally, our enhanced rendering selectively combines the two volumes to provide an responsive visualization. We leverage the instrument mask and projection images to combine I_c and I_a on the fly during rendering and adapt the rendering parameters, encoding the reliability of the information through color.

Fig. 2.

Our processing pipeline for a single new acquired volume. Reference projection maps are generated from an overview scan at acquisition start.

2.1. Axial Projection Images

Axial projection images are a class of 2D images that we generate from projecting every A-scan in the volume to a single corresponding value. Averaging projections have been used extensively in the past [13,14,6] as so-called Enface images. We generalize them to new functions which create 2D feature maps P(x, y) with different characteristics that we use for alignment and instrument segmentation:

$$P_{\mathrm{avg}}(x,y)=\frac{1}{N}\sum _{z}I(x,y,z),$$ (1a)

$$P_{\max }(x,y)=\underset{z}{\max }I(x,y,z),$$ (1b)

$$P_{\text{argmax}}(x,y)=\underset{z}{\arg \max }I(x,y,z),$$ (1c)

$$P_{\text{centr}}(x,y)=\sum _{z}zI(x,y,z)/\sum _{z}I(x,y,z)$$ (1d)

where N is the number of samples in one A-scan.

Note that while Average and Maximum encode intensity features, Argmax and Centroid provide positional information while they can still be computed simultaneously in a single iteration over the A-Scan voxels.

2.2. Learning-based instrument segmentation

A tissue mask M_t is obtained from the 2D projections by training a simple and lightweight segmentation model. We use the pytorch-based fast.ai library to train a UNet variant⁵ that uses a ResNet18-based encoder that was pretrained on ImageNet connected to a decoder with skip connections to perform subsequent upsampling. All four available projection images are provided as input features to the network to maximize accuracy.

2.3. Registration

To combine the sequence of volumes, it is required to know for every voxel the corresponding location in the compounded volume. During a surgery, the imaged region can change over time due to systematic changes of the galvo offsets, but also due to relative movement of the patient’s eye.

Motion Model.

With the eye anaesthetized and fixed in the eye socket, rotations generally only occur during brief, intentional adjustments of the view by the surgeon. During precise manipulation in which our OCT image guidance is used, the surgeon actively stabilizes the eye and thus we adjust one d.o.f. less than [14]. Due to our exponential averaging, only the most recent few volumes influence the displayed result. Analyzing DS3 (see Sec.3) using offline volumetric registration confirms that axial rotation is negligible in our data set as rotation between consecutive volumes is below 0.5°. We approximate the motion as a pure translational change in the OCT coordinate system. Two corresponding positions in reference and acquired volume p_r, p_a are related by p_r = p_a + t_a. We further decompose the translation t_a can be decomposed into the transverse translation (t_x, t_y) (mainly caused by rotational movement of the eye around its center) and the axial translation t_z (typically caused by inadvertent pressure on the trocars pushing the eye into or out of the eye socket).

Transverse Alignment.

Because the transverse alignment (t_x, t_y) is aligned with the layout of the generated projection images P_r, P_a, we can find the best alignment using template matching with normalized cross correlation (NCC) as a metric. To find the transverse alignment, we compute the location of maximum NCC between P_r and P_a using M_t to mask instrument pixels in the template.

Axial Alignment.

To find the axial alignment t_z, we use the P_argmax images in the region where the two images overlap based on (t_x, t_y). We compute the average distance across the overlap region O while taking into account the tissue mask M_t of the current image:

$$t_z=\sum _{p\in O}{M}_t(p)\left(P_{r,\mathrm{argmax}}(p)-P_{a,\mathrm{argmax}}(p)\right)/\sum _{p\in O}{M}_t(p)$$ (2)

2.4. Compounding

With a known transformation we can integrate the new volume I_a in our compounded representation I_c of the static scene:

$$I_c(p)=\omega (p)I_a\left(p+t_a\right)+(1-\omega (p))I_{\tilde{c}}(p)$$ (3)

for every voxel position p in I_c where ω = ω₀M_t(p) and ω₀ is a integration weighting parameter used to control the amount of exponential averaging over time. The update conditionally integrates new data for A-Scans where M_t is not set while retaining data from the previous compounded volume I_c˜ otherwise. Using the M_t in the integration weighting masks out A-scans containing the instrument while averaging the retinal tissue.

2.5. Rendering

To provide a well interpretable rendering that makes the user apparent of the reliability of the shown data, we use an adaptive colorization that extends the color map by Weiss [17]. Our goal is to emphasize which parts of the volume are from the last acquired volume and visually set apart data that has been inpainted from previous data. Thus, we differentiate three semantic regions within our volumes: (a) data that has been recently updated or is sampled directly from I_a, (b) data that has been inpainted from the I_c with no correspondence in I_a and (c) areas around the instrument. When determining the sample intensity I_s for a raymarch step position p_s, we combine I_c and I_a using an instrument predicate κ_i and a recency predicate κ_r defined as:

$${\kappa }_i(p)=\neg M_t(p)\wedge \left|p_z-P_{\mathrm{ArgMax}}(p)\right|<d_i,$$ (4a)

$${\kappa }_r(p)=M_t(p)\wedge I_a(p)>I_c(p).$$ (4b)

In A-scans that contain an instrument, we combine the intensities of both volumes with a max operation. The max operation ensures visibility of both instrument and inpainted tissue in areas of signal loss:

$$I_s\left(p_s\right)=\{\begin{array}{ll}{I}_c\left(p_s\right)\hfill & \text{if}{\kappa }_s(p)\hfill \\ max\left(I_c\left(p_s\right),I_a\left(p_s\right)\right)\hfill & \text{otherwise.}\hfill \end{array}$$ (5)

The recency predicate therefore marks sampled locations in which I_s was influenced by the most recently acquired volume, either because it was incorporated into the compounding (Eq. 3) or used directly (Eq. 5). In tissue regions, we apply the same layer colorization C_LAB(I, δ^∗(p)) as [17]. We compensate for the lack of a layer segmentation by instead using P_argmax to align the axial color mapping to the retinal structure. Furthermore, the colorization is only applied for pixels that have not been inpainted from the compounded volume, using a grayscale colormap for this data instead. For instrument areas, we use a fixed color C_i to further enhance instrument visibility and contrast to the surrounding tissue.

$$C_s\left(p_s\right)=\{\begin{array}{ll}{C}_i\hfill & \text{if}{\kappa }_i\left(p_s\right)\hfill \\ \left(I_s,I_s,I_s\right)\hfill & \text{if}\neg {\kappa }_r\left(p_s\right)\hfill \\ C_{\text{LAB}}\left(I_s,\delta *(p)\right)\hfill & \text{otherwise.}\hfill \end{array}$$ (6)

3. Results

As state of the art 4D iOCT systems are not readily available, the three data sets used in this work were instead recorded with a RESCAN 700 (Carl Zeiss Meditec) iOCT system (27kHz A-Scan rate) which acquires volumes of 512×128×1024 resolution at non-interactive rates. DS1 consists of projection images from 605 retinal OCT scans containing a range of different instruments and situations in human or porcine eyes. We generated projection images as described in Sec. 2.1, resample them to 128×128px and manually labeled the instruments. DS2 consists of 3×3×2.9mm³ OCT volumes from 6 enucleated porcine eyes with removed anterior segment. On a fixated eye, a 6×6×2.9mm³ reference volume and grids of 5×5 volumes were acquired with 0.25mm offset in x/y dimensions by changing the OCT acquisition offset. In three eyes, a 23G vitreoretinal forceps was used, in the remaining three a 27G injection needle. DS3 is a sequence of 67 volumes from an enucleated porcine eye with intact anterior segment. The first volume is acquired at 6×6×2.9mm³ while the following are recorded with 3×3×2.9mm³ field of view. A surgical instrument (23G retinal forceps) was moved between acquisitions to create a simulated 4D OCT sequence that shows instrument and retina motion as described in Sec. 2.3.

Registration.

To evaluate our registration approach, we use the known relative offsets between volumes in DS2. We use the reference volume of each eye and register each of the 25 grid volumes to it using our approach. Figure 3 shows the registration error $e_r=\sqrt{d_x^2+d_y^2}$ of the offsets d_x, d_y after registration for different projection images. The results show that using M_t dramatically improves the registration. Overall the argmax image performed best with an average offset error of 0.0426mm, RMSE of 0.0583mm and median error of 0.025mm.

Fig. 3.

Registration error (in mm) without using a tool mask compared to the registration error using M_t.

Instrument Segmentation.

We train our segmentation model on DS1 with a batch size of 64 and random data split of 80% training and 20% validation set. To overcome the problems related to limited size of our data set and to enhance the generalization of our segmentation model we augment with vertical and horizontal flipping, random rotation up to 10 degrees and random cropping to zoom up to a maximum scaling factor of 1.5. We use cross-entropy as our loss function and train with a learning rate of 0.01 and the AdamW [11] optimizer (β₁ = 0.9, β₂ = 0.99). Our model converged after 95 epochs and has a validation accuracy of 99.34% and 85.33% for retina and instrument classes, respectively.

Real-time processing.

Our real-time visualization system was implemented in C++. We use the OpenCL-accelerated template matching implementation of OpenCV and TensorRT⁶ to optimize and execute our trained model on the GPU. Projection image generation, volume compounding and rendering leverage the GPU using OpenGL. We evaluated computational performance by looping DS3 (see Supp. Video) with data preloaded to the GPU to simulate GPU reconstruction as in [9]. To compare our implementation to the data rates of a state of the art SS-OCT device, we resample our input data to a matching resolution of 330×330×595. Table 2 shows the average results on the original and resampled data sets on our evaluation system (Intel Core i7–8700K @3.7GHz, NVidia Titan Xp). The total processing time dictates both maximum frame rate rate of our setup and minimum latency from acquisition to display. Our achieved 21.32 frames per second is close to 24.2Hz volume rate rate reported by [9].

Table 2.

Average processing time in ms for volumes at source and resampled resolution.

	Project	Segment	Register	Compound	Render	Total	FPS
512×128×1024	1.92	7.41	11.91	7.61	19.00	47.85	20.90
330×330×595	1.81	6.68	20.80	8.49	9.13	46.91	21.32

4. Discussion and Conclusions

We presented the first end-to-end real-time pipeline to process and visualize 4D OCT data while mitigating instrument shadowing. Our adaptive visualization makes this processing and enhancement visible to the viewer and aids interpretation of the shown data. We demonstrated the feasibility of such a pipeline, achieving real-time processing speeds close to the state of the art imaging systems. Our evaluation reveals potential for future improvements for the registration approach. The implementation is based on off-the-shelf solutions for template matching and 2D image segmentation. Our matching errors are in the order of several px which leads to potentially oversmoothed results. Because the focus of this work was to introduce the general concept and benefits of temporal alignment in 4D OCT, we consider the evaluation and development of more robust registration solutions as future work. A further interesting challenge is how to mitigate deformations that are caused for example by tool-tissue interactions. With the current approach, these deformations are smoothed over time as the compounded volume I_c gets updated. A higher value for ω₀ can improve responsiveness in these cases at the cost of lower denoising, thus more fine-grained adaptations of ω(p) could be developed in future extensions.

4D iOCT is not yet widely available but has already attracted much interest within its community. In this work, we developed a novel visualization and tested it on simulated 4D iOCT on porcine eyes. The described approach is currently being integrated into a state-of-the-art 4D iOCT imaging system. Once the integrated system has received ethical approval, it will be able to supply data with moving instruments within human eyes and thus enable further investigations into the clinical benefits of our advanced processing and visualization.

Data Set and Code Availability

The data sets and code used in this study are available upon request from the corresponding author.

Table 1.

Comparison of projection images in terms of their registration performance using M_t (errors in mm).

Projection	Avg Error	RMSE	Std Dev
Average	0.5379	0.6743	0.6516
Maximum	0.1394	0.2034	0.3329
Argmax	0.0426	0.0583	0.0675
Centroid	0.2128	0.2985	0.2817