Abstract
Endoscopy is the gold standard for characterizing pediatric airway disorders, however it is limited for quantitative analysis due to lack of three-dimensional (3D) vision and poor stereotactic depth perception. We utilize structure from motion (SfM) photogrammetry, to reconstruct 3D surfaces of pathologic and healthy pediatric larynges from monocular two-dimensional (2D) endoscopy.
Models of pediatric subglottic stenosis were 3D printed and airway endoscopies were simulated. 3D surfaces were successfully reconstructed from endoscopic videos of all models using an SfM analysis toolkit. Average subglottic surface error between SfM reconstructed surfaces and 3D printed models was 0.65mm as measured by Modified Hausdorff Distance. Average volumetric similarity between SfM surfaces and printed models was 0.82 as measured by Jaccard Index.
SfM can be used to accurately reconstruct 3D surface renderings of the larynx from 2D endoscopy video. This technique has immense potential for use in quantitative analysis of airway geometry and virtual surgical planning.
Keywords: aerodigestive, computer vision, simulation, pediatric otolaryngology, subglottic stenosis
Introduction
Endoscopy is the gold standard technique for visualizing pediatric airway abnormalities, however it does not facilitate quantitative geometric analysis of laryngotracheal anatomy. Currently, quantitative characterization of airway anatomy is based on imaging including computed tomography (CT), however this method introduces radiation exposure, costs, and potential sedation needs.
Structure from motion (SfM) photogrammetry is an established computer vision algorithm which enables 3D reconstruction from a collection of two-dimensional (2D) images1. SfM has been applied to reconstruct gastric surfaces2 and to aid in camera localization in sinus and bladder procedures3,4. SfM has not previously been used for the 3D reconstruction of laryngotracheal surfaces from rigid monocular endoscopy, but it has great potential for airway morphometric analysis.
In this pilot study, we demonstrate a data collection pipeline that integrates with pediatric bronchoscopy and leverages SfM to create accurate laryngotracheal surface reconstructions and overcomes the scale ambiguity challenge of many 3D vision reconstruction algorithms5,6. This new tool can quantify airway dimensions and guide clinical decision making for patients with laryngotracheal abnormalities.
Methods
This study was approved by the Seattle Children’s Hospital Institutional Review Board (STUDY00004259).
A high-resolution CT scan of an infant with Grade 3 subglottic stenosis was segmented between the supraglottis and carina using 3DSlicer (slicer.org). The subglottis was manually expanded to 2mm, 3mm, and 4mm (Models A-C, respectively) to simulate differing degrees of airway constrictions. A second CT scan of a healthy two-year-old was also segmented (Model D). Models were 3D printed using Agilus and Verowhite via a Stratasys J750 printer. (Figure 1)
Figure 1:
Schematic depicting (A) representative sagittal slice of CT scan at level of larynx, (B) segmented airway lumen, (C) 3D-printed models using airway segmentations. The top panel represents subglottic stenosis (SGS) models while the bottom panel represents a healthy model.
Endoscopies were performed using Storz 2.7mm and 4.0mm 0-degree Hopkins telescopes, attached to a Stryker 1688 Camera Head and a Stryker 1288 HD Tower. Cameras were calibrated by recording images from multiple views of a 15x15mm planar checkerboard. Endoscopy was performed on each model by passing each endoscope through a Parsons 3 laryngoscope, through the stenosis if possible, distally to the carina. (Supplemental Figure) Video data was recorded at 30fps with 1280x1024 resolution.
3D airway surface reconstructions were performed using COLMAP1,7, an open-source SfM pipeline. Camera calibration parameters were calculated using OpenCV. The SfM algorithm was then used to extract scale invariant features from input images8, match features across frames, and estimate endoscope positions to triangulate surface features in a sparse 3D point cloud. Multi-view stereo (MVS) then generated a dense 3D point cloud7. A tetrahedral surface mesh was generated first using Delaunay triangulation, filtered using a Taubin filter, and remeshed using a Poisson meshing algorithm. The methodology for conversion of endoscopy videos into 3D mesh reconstructions is summarized in Figure 3.
Figure 3:
Flow diagram for conversion of endoscopy videos into 3D mesh reconstructions.
Each 3D reconstruction contained the distal laryngoscope. The physical dimensions of the laryngoscope were used to scale the SfM reconstructions in order to recover the absolute physical scale of the model (real-world units). SfM reconstructions are inherently scale ambiguous and distance measurements are arbitrary prior to scaling the model to real-world units. The SfM reconstructed surfaces were compared against the inner lumen geometries of the 3D printed airways, which were defined as “ground truth” models for comparison. The SfM reconstruction and ground truth surfaces were aligned with an iterative closest point algorithm. Model accuracy was determined using Modified Hausdorff Distance (MHD) and Jaccard Index (JI)9. MHD measures the similarity of two surfaces and is defined as the average of the minimum distance between each point in surface A and every point in surface B, while JI is a volumetric similarity metric defined as the ratio of the intersection and union of two volumes where a value of 0 represents entirely dissimilar objects, and 1 is identical. Reconstruction accuracy was evaluated on the entire SfM reconstructed airway and a subzone extending from the ventricles to the stenosis base.
Results
SfM-derived reconstructions were successfully generated for all models. Reconstructions of Model C and Model D captured the full length of the models. The stenoses of Model A and Model B were too narrow for the endoscope to pass through, so the SfM reconstructions of these models truncated distally to the subglottis (Figure 2). The average MHD for the SfM reconstructions was 0.59mm as compared to ground truth models while the average JI across the models was 0.83. The average MHD and JI for the glottis across models were 0.34mm and 0.83, respectively. (Table 1)
Figure 2:
Structure from motion generated reconstructions (teal) overlayed upon ground truth models (grey) in sagittal and coronal orientations.
Table 1:
Similarity between structure from motion (SfM) derived models and ground truth measurements by Modified Hausdorff Distance and Jaccard Index
| Full Reconstruction | Glottis Only | |||
|---|---|---|---|---|
| Modified Hausdorff Distance | Jaccard Index | Modified Hausdorff Distance | Jaccard Index | |
| Model A (2 mm) | 0.51 mm | 0.81 | 0.29 mm | 0.85 |
| Model B (3 mm) | 0.62 mm | 0.84 | 0.28 mm | 0.83 |
| Model C (4 mm) | 0.50 mm | 0.89 | 0.33 mm | 0.83 |
| Model D (Healthy) | 0.71 mm | 0.77 | 0.49 mm | 0.78 |
Discussion
Various computer vision techniques have been trialed in medical and surgical settings to reconstruct 3D surfaces of target organs including shape-from-shading (SfS), visual simultaneous localization and mapping (SLAM), and SfM10. The benefits of SfM include offline use, ability to process large, nonsequential image stacks, and high resolution reconstructions2. SfM has previously been used to reconstruct various internal organs from endoscopy including the sinus3, stomach2,6 and bladder.11 However, to our knowledge, SfM pipelines have not been used to reconstruct laryngotracheal surfaces.
Most 3D reconstruction methods, including SfM, suffer from scale ambiguity5. This limits the potential of SfM generated laryngotracheal surfaces for quantitative diagnostic purposes as no information on airway or stenosis diameter can be obtained without ground truth measurements.6,12,13. In this study we take advantage of the workflow of laryngoscopy and use the known physical scale of the laryngoscope to generate airway surfaces at the correct world scale.
In preliminary reconstructions, we have noticed loss of fidelity in areas of airway expansion, such as transitioning from feature-rich stenotic regions, to areas of higher luminal diameter. This does not pose an issue with reconstruction of the proximal laryngotracheal stenosis, but may introduce variance of airway caliber immediately distal of a narrowing.
In our pilot study, we reconstructed laryngotracheal surfaces with high accuracy, with an average MHD rivaling the pixel size of high-resolution CT scans. This methodology is easily implementable into standard operative workflows of airway endoscopies as the only added intraoperative step is to obtain a camera calibration (less than 30 seconds). This method can be used as an alternative tool for quantitative assessment of complex airways, allowing for improved understanding of airway morphometry in postoperative surveillance endoscopies and patients with indwelling devices such as endotracheal tubes which limits interpretability of airway CT scans. The tool also has potential for virtual surgical planning, simulation, and family counseling. SfM reconstructions can also be used in additional offline applications to better understand patient specific airflow via integration with computational fluid dynamics modeling, a unique interest of our research group.
Conclusion
We demonstrate a novel pipeline for creating accurate laryngotracheal surface reconstructions that can be quantified from bronchoscopy videos using SfM technology. Work is ongoing to validate the method in-vivo.
Supplementary Material
Supplemental Figure: Workflow of obtaining sufficient endoscopy videos for SfM reconstruction. (A) A 15mm x 15mm planar checkerboard was used to calibrate the optical properties of the endoscope. (B) A Parsons 3 laryngoscope was suspended over the 3D printed airway models to simulate operative endoscopy. (C) Representative photo of endoscope introduction through laryngoscope and into model. The dimensions of the laryngoscope are used as ground truth for airway measurements.
Funding:
This project was supported by the NIH/NIDCD T32 Research Training Grant DC000018 and Sie-Hatsukami Research Endowment Grant
Footnotes
Conflict of Interest: Dr. Bly is co-founder and holds a financial interest of ownership equity with Wavely Diagnostics Inc and Apertur Inc. He is a consultant and stockholder, Spiway LLC. These are not related to this study. All other authors do not have information to disclose.
Meeting Information: Accepted for oral presentation; American Academy of Otolaryngology-Head and Neck Surgery Foundation Annual Meeting; September 2023; Nashville, TN
Bibliography
- 1.Schonberger JL, Frahm JM. Structure-from-Motion Revisited. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE; 2016:4104–4113. doi: 10.1109/CVPR.2016.445 [DOI] [Google Scholar]
- 2.Widya AR, Monno Y, Okutomi M, Suzuki S, Gotoda T, Miki K. Whole Stomach 3D Reconstruction and Frame Localization From Monocular Endoscope Video. IEEE J Transl Eng Health Med. 2019;7:1–10. doi: 10.1109/JTEHM.2019.2946802 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Leonard S, Sinha A, Reiter A, et al. Evaluation and Stability Analysis of Video-Based Navigation System for Functional Endoscopic Sinus Surgery on In Vivo Clinical Data. IEEE Trans Med Imaging. 2018;37(10):2185–2195. doi: 10.1109/TMI.2018.2833868 [DOI] [PubMed] [Google Scholar]
- 4.Gong C, Zhou Y, Lewis A, et al. Real-Time Camera Localization during Robot-Assisted Telecystoscopy for Bladder Cancer Surveillance. J Med Robot Res. 2022;07(02n03):2241002. doi: 10.1142/S2424905X22410021 [DOI] [Google Scholar]
- 5.Hartley RI, Zisserman A. Multiple View Geometry in Computer Vision. Second. Cambridge University Press, ISBN: 0521540518; 2004. [Google Scholar]
- 6.Alcantarilla PF, Bartoli A, Chadebecq F, Tilmant C, Lepilliez V. Enhanced imaging colonoscopy facilitates dense motion-based 3D reconstruction. In: 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE; 2013:7346–7349. doi: 10.1109/EMBC.2013.6611255 [DOI] [PubMed] [Google Scholar]
- 7.Schönberger JL, Zheng E, Frahm JM, Pollefeys M. Pixelwise View Selection for Unstructured Multi-View Stereo. In: Computer Vision–ECCV 2016: 14th European Conference. Vol 9907. Springer International Publishing; 2016:501–518. doi: 10.1007/978-3-319-46487-9_31 [DOI] [Google Scholar]
- 8.Lowe DG. Distinctive Image Features from Scale-Invariant Keypoints. Int J Comput Vis. 2004;60(2):91–110. doi: 10.1023/B:VISI.0000029664.99615.94 [DOI] [Google Scholar]
- 9.Taha AA, Hanbury A. Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool. BMC Med Imaging. 2015;15(1):29. doi: 10.1186/s12880-015-0068-x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Bergen T, Wittenberg T. Stitching and Surface Reconstruction From Endoscopic Image Sequences: A Review of Applications and Methods. IEEE J Biomed Health Inform. 2016;20(1):304–321. doi: 10.1109/JBHI.2014.2384134 [DOI] [PubMed] [Google Scholar]
- 11.Péntek Q, Hein S, Miernik A, Reiterer A. Image-based 3D surface approximation of the bladder using structure-from-motion for enhanced cystoscopy based on phantom data. Biomed Eng Biomed Tech. 2018;63(4):461–466. doi: 10.1515/bmt-2016-0185 [DOI] [PubMed] [Google Scholar]
- 12.Chadebecq F, Tilmant C, Bartoli A. Measuring the size of neoplasia in colonoscopy using depth-from-defocus. In: 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE; 2012:1478–1481. doi: 10.1109/EMBC.2012.6346220 [DOI] [PubMed] [Google Scholar]
- 13.Liu X, Sinha A, Ishii M, et al. Dense Depth Estimation in Monocular Endoscopy With Self-Supervised Learning Methods. IEEE Trans Med Imaging. 2020;39(5):1438–1447. doi: 10.1109/TMI.2019.2950936 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Supplemental Figure: Workflow of obtaining sufficient endoscopy videos for SfM reconstruction. (A) A 15mm x 15mm planar checkerboard was used to calibrate the optical properties of the endoscope. (B) A Parsons 3 laryngoscope was suspended over the 3D printed airway models to simulate operative endoscopy. (C) Representative photo of endoscope introduction through laryngoscope and into model. The dimensions of the laryngoscope are used as ground truth for airway measurements.