High-Fidelity 3D Reconstruction for Accurate Anatomical Measurements in Endoscopic Sinus Surgery

Abstract

Achieving an accurate representation of the surgical scene is essential, as it enables precise surgical navigation. Surgeons currently rely on preoperative computed tomography (CT) scans to represent the surgical scene and plan sinus procedures. However, as tissue is resected and manipulated, the anatomy represented in preoperative images becomes increasingly inaccurate and outdated. The endoscopic 3D reconstruction provides an alternative solution to this challenge, for it captures the current surgical scene. Nevertheless, achieving high reconstruction accuracy is crucial in endoscopic sinus surgery (ESS), where tissue margins lie within submillimeter distances to critical anatomy such as the orbits, cranial nerves, carotid arteries, and dura mater. To fulfill the need for a highly accurate intraoperative method of surgical scene modeling in ESS, we propose a system to generate 3D reconstructions of the sinus to garner relevant qualitative and quantitative anatomic information that substantially diverges from preoperative CT images as the surgery progresses. To achieve this, the pipeline of Neural Radiance Fields (NeRF) is expanded to include methods that simulate stereoscopic views using only a monocular endoscope to iteratively refine the depth of reconstructions. The presented workflow provides accurate depth maps, global scaling, and geometric information without camera pose-tracking tools or fiducial markers. Additional methods of point cloud denoising, outlier removal, and dropout patching have been developed and implemented to increase point cloud robustness. This expanded workflow demonstrates the ability to create high-resolution and accurate 3D reconstructions of the surgical scene. Using a series of three cadaveric specimens, measurements of critical anatomy were evaluated with average reconstruction errors for ethmoid length and height being 0.25mm and 0.52mm, respectively.

1. Introduction

1.1. Endoscopic Sinus Surgery

The need for accurate tissue margin modeling is especially critical in endoscopic sinus surgery (ESS), which necessitates the complete removal of diseased tissue, bone, and cartilage to widen drainage and airway passages within the sinus, often within submillimeter margins of critical anatomy. Nearly 300,000 ESS procedures are performed annually in the United States, with 1–3% resulting in damage to critical structures [1, 2]. The risk of such damage often leads surgeons to be overly conservative in tissue resection, causing incomplete removal of diseased tissue and necessitating correctional procedures in 6–28% of cases [3, 4]. These critical damages could be significantly reduced through accurate nasal cavity modeling combined with surgical navigation. While preoperative CT is currently the most widely used technique for tissue margin modeling, the anatomical model becomes increasingly inaccurate as tissue is resected during surgery. Although intraoperative CT could provide updated anatomical information, it is not widely adopted due to extended surgical time that increases intraoperative risk, additional economic costs, and radiation exposure to both patients and surgeons [5, 6]. Therefore, developing a high-accuracy, safe method for intraoperative surgical scene modeling is of paramount importance.

1.2. Endoscopic 3D reconstruction

To solve the problem mentioned above, the monocular endoscopic 3D reconstruction presents a promising alternative to CT, as it can capture and reconstruct the current surgical scene in three dimensions without the safety concerns and financial burden associated with intraoperative CT. However, existing reconstruction techniques face several challenges. Structure-from-motion (SfM) based methods can achieve accuracy within 1–2 millimeters but suffer from slow processing speeds [7–9]. Simultaneous Localization and Mapping (SLAM) based approaches offer real-time performance but often lack the required accuracy [10–14]. Deep learning-based depth estimation methods face generalization issues, where performance deteriorates significantly when encountering data dissimilar to their training sets [15]. This is particularly problematic in ESS procedures where limited datasets are available, resulting in reconstruction accuracy that falls short of SfM-based methods [7].

1.3. Neural Radiance Fields

Neural Radiance Fields (NeRF) [16] as a newly developed neural rendering technique demonstrate powerful representation capabilities for complex scenes, it takes a set of input images $I=\left\{I_1,I_2,I_3,\dots \right\}$ and their 5D camera poses (spatial coordinates $\left(p_x,p_y,p_z\right)$ and viewing direction $(\theta ,\varphi )$), and outputs volume density and radiance to reconstruct high-fidelity scenes via volume rendering. Unlike feature-based methods (such as traditional SfM and SLAM) and pre-trained deep learning approaches, NeRF does not rely on feature point matching, thus avoiding the “low-feature region reconstruction” problem, nor does it require a pre-trained model from training datasets, eliminating generalization issues caused by dataset variations. By directly optimizing from 5D input data, this distinctive characteristic makes NeRF one of the most promising approaches for 3D reconstruction. In recent research, this advanced technique has shown promising initial results in medical applications, demonstrating superior performance in several preliminary studies [17–20]. However, in surgical environments, issues such as variable lighting conditions, low texture, repetitive tissue patterns, obscured features, and low-contrast scenes can lead to poor-quality point clouds and meshes, often characterized by frequent holes, dropouts, or inaccurate textures where features were inadequately imaged.

In this work, we address these challenges through our novel hybrid NeRF-stereo implementation, enhanced with specialized point cloud tuning and mesh extraction methods, to improve the accuracy of surgical field modeling.

2. Methods

2.1. NeRF Deployment

Building upon our previous work [21, 22], we implement this method using NeRFstudio [23], a modular PyTorch framework that provides plug-and-play components for NeRF development. This platform choice enables real-time visualization capabilities and convenient data conversion between various formats including video, point clouds, and meshes, making our method more accessible to end users.

In this work, as illustrated in Figure 1, we create an initial NeRF reconstruction that processes 5D input vectors combining spatial coordinates and viewing directions. Specifically, given a set of camera poses (obtained from COLMAP) $\left\{{\mathbf{p}}_1,{\mathbf{p}}_2,\dots ,{\mathbf{p}}_n\right\}$, where ${\mathbf{p}}_i=\left(p_{x_i},p_{y_i},p_{z_i}\right)$ represents the spatial location, and viewing directions $\mathbf{d}=\left\{\left({\theta }_1,{\varphi }_1\right),\dots ,\left({\theta }_n,{\varphi }_n\right)\right\}$, where ${\mathbf{d}}_i=\left({\theta }_i,{\varphi }_i\right)$ denotes the spherical coordinates of the viewing direction, our system develops conical frustums for each pixel and reconstructs the volume with color and density values [16]. Through this volumetric rendering process, we obtain a function:

$$F\left({\mathbf{p}}_j,{\mathbf{d}}_j\right)=\text{NeRF}\left(p_{x_j},p_{y_j},p_{z_j},{\theta }_j,{\varphi }_j\right)=I_j$$ (1)

where ${\mathbf{p}}_j,{\mathbf{d}}_j$ represent arbitrary camera pose and viewing direction, and $I_j$ is the rendered image from this viewpoint. Nerfacto [23] is the NeRF method we used in this work.Then, the camera pose matrix $M_j=\left[R_j\mid t_j\right]$ can be derived from ${\mathbf{p}}_j$ and ${\mathbf{d}}_j$ using Rodrigues’ formula. After training our NeRF function $f$ that maps camera poses to images, we proceed to generate stereo pairs for depth estimation. For each camera pose $M_i=\left[R_i\mid t_i\right]$ in our training set, we create a novel stereo configuration by defining extrinsic parameters $S=\left[\begin{array}{ll}E& \mathbf{b}\end{array}\right]$, where $E\in {\mathbf{R}}^{3\times 3}$ is an identity matrix, and $\mathbf{b}=(b,0,0{)}^T$ represents a translation vector with magnitude $b$ along the $x$-axis in the camera reference system.

Figure 1:

Method and workflow for NeRF implementation, utilizing simulated stereoscopic views to iteratively refine the reconstruction.

For each training viewpoint, we generate a stereo pair using our trained NeRF by:

1. Rendering the left (reference) image using the original pose:

   $$I_i^L=F\left(M_i\right)$$ (2)
2. Computing the novel right camera pose and rendering the corresponding right image:

   $$M_i^R=M_i\times S=\left[R_i\mid \left(t_i+\mathbf{b}\right)\right]$$ (3)

   $$I_i^R=F\left(M_i^R\right)$$ (4)

The resulting images form a stereo pair with the target frame positioned to the right of the reference frame, enabling the generation of multiple stereo samples for depth estimation.

Then, we utilize stereo disparity estimation to compute the depth $z$ using:

$$z_i=\frac{fb}{{\text{disp}}_i},$$ (5)

where $f$ is the focal length, $b$ is the baseline length between the stereo cameras, and disp is the disparity value obtained from the stereo image pair.

The disparity is computed using a disparity estimation algorithm $D$ applied to the left and right images $I_i^L$ and $I_i^R$:

$${\text{disp}}_i=D\left(I_i^L,I_i^R\right).$$ (6)

Selective-stereo [24] are used as $D$. After obtaining the depth map from the initial NeRF reconstruction via stereo disparity estimation, we use this depth to supervise and refine the NeRF model. Specifically, we calculate the depth supervision loss following DS-NeRF [25]. For each image $I_j$ and camera pose ${\mathbf{p}}_j$, the initial depth map $z_{{\text{init}}_j}$ is used to supervise the NeRF model.

We model the depth estimation noise by treating the observed depth $z_{ij}^{\prime }$ at each pixel as a Gaussian distribution with variance ${\hat{\sigma }}_i^2$, i.e., $z_{ij}^{\prime }~𝒩\left(z_{ij}^{\prime },{\hat{\sigma }}_i^2\right)$. The depth supervision loss ${ℒ}_{\text{Depth}}$ is defined by minimizing the KL divergence between the predicted depth distribution $h\left(t\right)$ from NeRF and the observed depth distribution:

$${ℒ}_{\text{Depth}}={\mathbb{E}}_{z_i\in Z_j}\left[\int h(t)\text{log}h(t)\text{exp}\left(-\frac{{\left(t-z_{ij}^{\prime }\right)}^2}{2{\hat{\sigma }}_i^2}\right)dt\right]\approx {\mathbb{E}}_{z_i\in Z_j}\left[\sum _k{h}_k\text{log}{h}_k\text{exp}\left(-\frac{{\left(t_k-z_{ij}^{\prime }\right)}^2}{2{\hat{\sigma }}_i^2}\right)\Delta t_k\right].$$ (7)

The overall loss function for NeRF training combines the photometric loss and the depth supervision loss:

$$ℒ={ℒ}_{\text{Images}}+{\lambda }_z{ℒ}_{\text{Depth}},$$ (8)

where ${ℒ}_{\text{Images}}$ is the loss from NeRF training process, and ${\lambda }_z$ is a weighting factor balancing the two losses.

This process iteratively refines NeRF: after each iteration, the updated NeRF model produces new depth maps, which are then used to supervise the next iteration. The iteration continues until convergence, defined by:

$$\left|1-\left|\frac{Z_{\text{iter}}}{Z_{\text{iter}-1}}\right|\right|<\epsilon ,$$ (9)

where $Z_{\text{iter}}=\left\{z_{{\text{iter}}_1},z_{{\text{iter}}_2},\dots ,z_{{\text{iter}}_n}\right\}$ and $Z_{\text{iter}-1}=\left\{z_{\text{iter}-1_1},z_{\text{iter}-1_2},\dots ,z_{\text{iter}-1_n}\right\}$ are sets of depth values from the current and previous iterations respectively, and $\epsilon$ is a small threshold. Through this iterative process, we obtain a high-quality NeRF reconstruction and an optimized depth map set $Z_{\text{final}}=\left\{z_{{\text{final}}_1},z_{{\text{final}}_2},\dots ,z_{{\text{final}}_n}\right\}$.

2.2. Point Cloud & Mesh Extraction

Beyond achieving accurate NeRF scene representation, mesh extraction presents another critical challenge in NeRF-based 3D reconstruction. The inherent nature of NeRF’s volumetric representation, which consists of discrete voxels in space, makes high-quality mesh extraction particularly challenging. Although NeRFStudio includes built-in functionality for mesh extraction, it is not inherently designed to export point clouds and meshes for measurement extraction and evaluation [26]. Standard meshes extracted from NeRF reconstructions using Poisson surface reconstruction can exhibit surface dropouts, noise, and outliers. To address these issues, we emploied PointCleanNet [27] a self-tuning learning-based algorithm that effectively balances noise reduction with feature preservation. It effectively preserves areas of the input that do not require denoising or outlier removal. The architecture, built upon PCPNet, estimates local features within the scene, such as surface normals and curvature [28]. Leveraging these estimates, PointCleanNet labels and removes outlier points, estimates correction vectors, and projects noisy points to their correct positions on the cleaned surfaces.

While PointCleanNet effectively denoises and removes outliers from reconstructed point clouds, it does not address dropouts or sparse regions. To tackle these issues, we have developed a point cloud upsampling and patching method that takes the cleaned point cloud from PointCleanNet as input for further processing. This method evaluates the point cloud’s distribution to identify and analyze sparse regions. The cleaned point cloud and training parameters serve as a template for placing additional points.

The process begins with propagating a low-subdivision octree throughout the scene, where each leaf node stores the local point density. This density is assessed using dynamic thresholding to guide upsampling, with a focus on sparse regions representative of the ground truth anatomy while avoiding unoccupied areas. The octree is further subdivided based on a threshold value, $\nu$, which determines whether a node qualifies for further subdivision, and therefore upsampling. If the point count in a node falls below $\nu$, it is deemed unoccupied by ground truth anatomy and excluded from further subdivision. Nodes exceeding $\nu$ are subdivided further, with each new level of the octree subjected to updated $\nu$ thresholds, reflective of expected density at that resolution. By traversing the octree, we can query density information across spatial partitions, maintaining high computational efficiency through adaptive resolution. Once the octree is fully subdivided and density values are assigned to each voxel, upsampling is performed where needed. Qualified voxels are populated using Poisson Disk Sampling, which generates tightly packed points with a user-specified minimum spacing, ensuring a uniform yet natural distribution. After upsampling, a second Poisson surface reconstruction is applied to create an enhanced mesh. This final step minimizes outliers, noise, dropouts, and holes while preserving the original anatomy’s integrity and specificity. The results of point cloud upsampling are shown in Figure 2.

Figure 2:

Results of point cloud upsampling via multi-resolution octree subdivision and dynamic thresholding for Poisson Disk Sampling.

2.3. Scaling

Various monocular endoscopic reconstruction methods have proven effective in extracting measurements with high relative accuracy or simulating absolute accuracy by scaling reconstructions to match ground truth CT scans. However, standard NeRF implementations are not globally accurate or scaled, as they rely on locally estimated camera poses obtained by pairing sequential images for incremental triangulation [16]. The resulting point coordinates are normalized and remapped into [−1,1], centering the ray origins in the scene and bounding coordinates within a cube at the origin. While SLAM methods use techniques such as loop closure and pose graph optimization to improve global accuracy, NeRF reconstructions cannot be scaled accurately without additional tools such as SLAM or LiDAR for pose scaling [29–32]. Our implementation of the hybrid NeRF-stereo method is able to extract globally accurate metric depth when camera poses are accurately scaled through the use of electromagnetic endoscopic pose tracking systems. Through, to achieve absolute scaling accuracy without relying on pose tracking systems or a ground truth CT, we incorporate a small printed pattern with known dimensions into the scene, in this case, a checkerboard pattern with square side lengths of 1.25mm. Adequate views of the pattern are captured and reconstructed alongside the rest of the surgical scene and are subsequently scaled based on the known dimensions, ensuring consistent and highly accurate scaling when pose tracking systems are not available.

3. Experiments

3.1. Cadaver Method

ESS was performed on a series of three cadaveric specimens. Complete endoscopic sinus surgery, including bilateral maxillary antrostomies, total ethmoidectomies, sphenoidotomies, and frontal sinusotomies. The surgeries were recorded using a high definition endoscope: the Image 1S 4U 2D camera head, with a 0 degree 4mm monocular rigid endoscope, and a Storz 4K AIDA video management unit. The high definition video was used as the sole input to the reconstruction pipeline (Figure 3). The cadaveric specimens underwent postoperative CT imaging using a Philips Somatom Edge scanner (slice thickness: 0.625 mm). The point clouds generated from the reconstruction were registered to the CT scans within 3DSlicer through the manual matching of distinct anatomic features, which was overseen by resident otolaryngologist.

Figure 3:

(A) Qualitative results of the surgical scene reconstruction, demonstrating endoscopic video frames and their respective comparisons to reconstructed anatomical structures. (B) Measurements taken of ethmoid length and height on 3 cadaveric specimens from two observers, including an ESS surgeron. (C) Visualization of endoscopic length and height in comparison to bony anatomy as seen on the postoperative CT scan.

The NeRF training and meshing training were conducted on a workstation equipped with an AMD Ryzen^™ Threadripper^™ 1950X processor and an NVIDIA RTX 4080 GPU.

3.2. Measurement Method

For clinical applicability, it is essential that reconstructions not only capture the scene with absolute accuracy but also precisely represent the measurements of vital anatomy compared to ground truth CT image set. Two measurements of the ethmoid cavity—the Ethmoid Length and Ethmoid Height—were selected for their clinical relevance, proximity to critical structures, and correlation with fixed bony regions that are distinct on CT images and can be repeatedly measured by an ESS surgeon. These measurements define a space that is typically completely resected during surgery to remove inflammatory tissue and restore natural sinus drainage. The ethmoid sinus is located only a few millimeters inferior to the skull base and brain, and medial to the lamina papyracea, a thin bone separating the nasal cavity from the eye and optic nerve. The Ethmoid Length is defined as the distance between the posterior surface of the frontal bone and the anterior face of the sphenoid rostrum. The Ethmoid Height is the distance between the floor of the sphenoid sinus at the level of the sphenoid ostium and the ethmoid skull base. Measurements of these structures on the CT scan were compared with measurements from the reconstruction, as shown in Figure 3B, with a visual demonstration of ethmoid sinus length and height in Figure 3C. Additionally, the qualitative results of these reconstructions are presented in Figure 3A.

4. Discussion

To our knowledge, this study is the first to apply a NeRF reconstruction method to generate 3D models for extracting clinically relevant anatomical measurements crucial for ESS. Previous ESS reconstruction methods have focused on general volumetric comparisons or subjective tissue and bony landmarks, which are less commonly used in clinical practice [33–36]. Unlike comparable methods, this study does not rely on external tracking tools for camera pose acquisition, nor does it use CT images or additional tools for global scaling of reconstructions [35,36]. Additionally, our workflow does not require fiducial markers for high-accuracy reconstructions and registration, relying only on natural tissue fiducials for alignment. This method employs a monocular endoscope, a tool used in all ESS, to achieve high-resolution and accurate reconstructions. It has demonstrated consistent submillimeter accuracy, with an average reconstruction error of 0.52 mm (± 0.55 SEM) and 2.46% for the distance from the floor of the sphenoid sinus in the plane of the sphenoid ostium to the ethmoid skull base, and 0.25 mm (± 0.61) and 0.84% for the length from the frontal bone to the face of the sphenoid rostrum. These results improve upon the accepted accuracy range of 2mm or less for clinical usefulness in ESS navigation and match or surpass the performance of state-of-the-art methods, some of which use external tools or rely on preoperative CT data for improved quality, navigation, pose acquisition, scaling, or registration—all of which are avoided in our method [33–37].

Current work aims to address the remaining challenges in surgical scene reconstruction. While our workflow eliminates the necessity for pose tracking systems, stereoscopes, and fiducial markers for reconstruction viability, it still requires initial manual coregistration of natural tissue fiducials, which would cause challenges in certain clinical applications. We anticipate that registration could be furthered past manual techniques by specializing comparable methods typically used for deformable and dynamic MRI co-registration, such as the Dense Correspondence Analysis (DeCA) toolkit, and the Advanced Normalization Tools (ANTs) suite. DeCA is an open-source toolkit primarily used by biologists for the initial labeleling, matching, and coregistration of anatomical and structural landmakrs through highly detailed shape analysis, which is suitable for smooth, sparse, and even deformable landmarks. ANTs could then provide affine registration (scaling and shear), and nonlinear deformable registration (local contraction and expansion) after an initial landmark-based registration using DeCA, which would better capture fine-grained anatomical coorespondance. An automatic registration workflow would make methods of quantitative reconstruction more streamlined and rapid in progression, and therefore more suitable for clinical application. Additionally, while using cadaveric specimens provides a relevant anatomical model, it does not account for blood accumulation and surgical tool occlusion. Our team has previously studied ESS medical tool segmentation and removal of irrelevant or degraded frames from surgical videos and developed a comprehensive labeled dataset to assist in this process [38–40]. We will leverage this prior work to address issues of surgical tools and blood occlusion in endoscopic videos. Reducing computation time is also an area of active research, as reconstruction convergence does not occur in real-time, as is desired for clinical use.

Neural radiance fields have only recently been explored for quantitative measurement extraction, the processes for which are still in active development [17, 31]. Quantitative NeRF methods are seen to be contained to real-world mapping tasks, comparing NeRF reconstructions to LiDAR techniques on a centimeter, rather than millimeter, scale.

5. Conclusion

The implementation of these methods has the potential to impact multiple surgical diciplines reaching past endoscopic sinus surgery, and, in the future, could be applied to a broad range of other minimally invasive surgical disciplines. These methods could allow for navigation in environments which present prevalent visually obscured or deformable anatomy, the visual representation of which could be compensated for with the presented computer vision and reconstruction techniques which are not currently possible with conventional IGS. The work presented here highlights NeRF’s ability to create high-fidelity, photorealistic visualizations of surgical scenes and for extracting globally accurate, point-to-point anatomical measurements using only a monocular endoscope.

This work has not and will not be submitted for publication or presentation elsewhere.