Abstract
There has been a long pursuit for precise and reproducible glomerular quantification in the field of renal pathology in both research and clinical practice. Currently, 3D glomerular identification and reconstruction of large-scale glomeruli are labor-intensive tasks, and time-consuming by manual analysis on whole slide imaging (WSI) in 2D serial sectioning representation. The accuracy of serial section analysis is also limited in the 2D serial context. Moreover, there are no approaches to present 3D glomerular visualization for human examination (volume calculation, 3D phenotype analysis, etc.). In this paper, we introduce an end-to-end holistic deep-learning-based method that achieves automatic detection, segmentation and multi-object tracking (MOT) of individual glomeruli with large-scale glomerular-registered assessment in a 3D context on WSIs. The high-resolution WSIs are the inputs, while the outputs are the 3D glomerular reconstruction and volume estimation. This pipeline achieves 81.8 in IDF1 and 69.1 in MOTA as MOT performance, while the proposed volume estimation achieves 0.84 Spearman correlation coefficient with manual annotation. The end-to-end MAP3D+ pipeline provides an approach for extensive 3D glomerular reconstruction and volume quantification from 2D serial section WSIs.
Keywords: 3D reconstruction, registration, segmentation, MOT
1. INTRODUCTION
Modern pathology analysis is shifting towards digital image analysis with a computer assistant to view scanned histology slides, which provides tremendous advances on whole slide imaging (WSI).1 These advances are mainly attributed to development in deep learning approaches. The rapid growth of deep learning technologies has enabled enormous amounts of glomerular quantification in large-scale high-resolution renal pathology images for clinical research and practice. Glomerular lesions, which can be captured by 3D glomerular voxel, can give significant information for the classification of disease processes with importance for response to treatment or progression of disease. However, current deep learning applications of glomerular quantification are primarily employed on two-dimensional (2D) WSI serial sections, which lack robustness and reproducibility of 3D context.2 Therefore, 3D glomerular identification and reconstruction of large-scale glomeruli are still labor-intensive tasks, and time-consuming tasks by a manual assessment on WSI in 2D serial sectioning representation with limited accuracy.
The explosive progress in deep learning technologies has been employed for glomerular quantification in 2D WSIs, which include glomerular detection,3 segmentation4,5 and classification.6 These methods can separately meet specific analysis demands in renal pathology, but fail to associate individual glomerular identification from sectioning slides. Our previous AI approaches Map3D7 performed registration-based Multi-Object Tracking (MOT) to automatically achieve 3D identification and the association of glomeruli from 2D serial WSIs. However, there are still no approaches to present 3D glomerular reconstruction and quantification for human examination (volume calculation, 3D phenotype analysis, etc.).
Herein, we introduce an end-to-end holistic deep-learning-based method Map3D+ that achieves semantic segmentation, patch-wise registration, and association algorithm for each glomerulus in WSIs (Figure 1). The high-resolution WSIs are the inputs, while the outputs are the 3D glomerular representations with segmentation and volume estimation. The proposed pipeline achieves automatic detection, segmentation, and multi-object tracking (MOT) of individual glomeruli with large-scale glomerular-registered assessment in a 3D context.
Figure 1.
This figure shows the overall motivation of this study, which is to provide an end-to-end holistic deep-learning-based method to achieve 3D glomerular reconstruction and quantification from 2D serial WSIs.
Briefly, the novelties of the proposed approach are four-fold: (1) A novel end-to-end holistic deep-learning-based pipeline is proposed to automatically achieve large-scale 3D glomerular reconstruction and quantification for human examination; (2) A multi-modality patch-wise registration is presented to precisely reconstruct 3D glomerular visualization; (3) A segmentation-based tracking adjustment is introduced to improve MOT performances; (4) A segmentation-based volume estimation is employed to provide individual glomerular voxel in 3D representation.
2. METHOD
The overall end-to-end Registration-based multi-object 3D reconstruction Map3D+ is presented in Figure 2. The Map3D+ pipeline consists of three sections:(1) Scan-wise global glomerular identification, (2) Multi-modality patch-wise registration and association, and (3) Segmentation-based 3D glomerular reconstruction and quantification.
Figure 2.
This figure presents the whole framework of Map3D+. The end-to-end pipeline is divided into 3 steps: (1)Scan-wise global glomerular identification, (2) Multi-modality patch-wise registration and association, and (3) Segmentation-based 3D glomerular reconstruction and quantification.
2.1. Scan-wise global glomerular identification
The scan-wise global glomerular identification is succeeded by a glomerular detection and a registration-based Multi-Object 3D association. First, our previously proposed CircleNet approach8 is implemented on WSIs to achieve large-scale glomerular detection. All the detection outcomes are stored as bounding boxes with their corner coordinates for the following association. A Graph Neural Network-based keypoint registration Super-Glue(SG)9 is employed to succeed the whole series affine registration on WSIs. All the sections are transformed to the physical space of the middle section by transformation propagation. After receiving all of the detection results and the global affine registration transformation, our previous Dual-path Association (DPA) from Map3D is implemented to track all the detection objects across serial sections by measuring the similarity score Intersection over Union (IoU).
2.2. Multi-modality patch-wise registration and association
Due to imaging artifacts and missing tissues, the global affine registration on WSIs is deficient for precise glomerular identification and association. To provide robust registration for 3D glomerular reconstruction and assessment, we introduce a multi-modality patch-wise registration and association for glomeruli.
For each global glomerular association result from Section 2.1, the bounding box coordinate whose section is closest to the middle section is selected and transformed to the physical space of the middle section as a middle section bounding box. All middle section bounding boxes are regarded as centers to collect sectioning patches in the entire transformed series. For all the patch sequences, a pretrained semantic segmentation model DeepLabV3 (Resnet-50)10 is employed to achieve glomerular binary segmentation on each patch.
Because of the imperfect global affine registration, there are numerous large displacements on glomerular sectioning patches, which is challenging for intensity-based pathology registration. Therefore, a multi-modality registration method inspired by AirLab11 is proposed to utilize both intensity and segmentation patches in order to obtain robust patch-wise registration. The segmentation patches can resolve large displacements, while the intensity patches can accurately align glomerular with adjacent tissue information.
Once achieving all of the patch-wise registration, two MOT adjustment algorithms are implemented on former assessment results from Section 2.1. The False-Positive Elimination (FPE) algorithm is used to remove redundant detection objects from CircleNet, determined by segmentation results in corresponding bounding box areas. The FPE algorithm is employed to the ends of series identification or non-continuous identifications, since the false-positive objects are concentrated on these intervals. The Identification Merging (IM) algorithm is used to combine two association sequences that are discontinued by deficient global registration. When there are missing tissues, the IM algorithm is implemented on two sections ahead of the section where each glomerulus appears and two sections after when each glomerulus disappears. The FPE algorithm is defined in Algorithm 1, while the IM algorithm is defined in Algorithm 2. An optimal threshold S is used in two algorithms to enhance the association robustness.
2.3. Segmentation-based 3D glomerular reconstruction and quantification
After precisely associating 3D glomerular identification with accurate patch-wise spatial registration, one segmentation-based mask selection algorithm (SBMS) is implemented to determine individual glomerulus segmentation on a single segmentation patch in each glomerular sequence. The SBMS algorithm is defined in Algorithm 3. A segmentation-based voxel estimation is then proposed to calculate the single glomerulus volume as
| (1) |
One common voxel estimation algorithm - Maximal profile area (MPA) volume12 - is implemented to compare the calculation performances. As a whole, a 3D stacking glomerular reconstruction is offered on the WSI for human phenotype analysis.
3. EXPERIMENTS
3.1. Data
12 mouse kidney sections (3D volumes) have been digitized from two previous studies*,†. 11 of them are used in the segmentation training processes in our study, while 1 kidney section is evaluated for the testing results. All animal procedures were approved by the Institutional Animal Care and the Use Committee at Vanderbilt University Medical Center. Each mouse kidney was prepared through paraformaldehyde fixation, paraffin embedding 7 to 17 8 μm thick sections were then cut, followed by deparaffinization and staining with hematoxylin for detection of nuclei and the Lotus tetragonolobus lectin for proximal tubule detection. The WSIs for all sections were scanned at 20× magnification (0.5 μm pixel resolution) with a Leica SCN400 Slide Scanner. Images were saved as .scn files.
3.2. Experiment Details
In global glomerular identification, we directly fine-tuned the previous CircleNet model for human glomeruli using cropped patches from 927 and 125 mice glomeruli as training and validation data, respectively. The official SG pretrained model is used for global affine registration.
In patch-wise segmentation, 17,495 patches (512 × 512) from 11 mouse kidney sections were used to train the DeepLab v3 model to obtain semantic segmentation for all glomerulus patches on the testing set. In multi-modality registration, 1000 × 1000 patches were cropped in each identification sequence. Two-steps registration strategies with 200 iterations were deployed on patch pairs. Structural Similarity loss (SSIM) of both intensity images and segmentation images, and the Normalized Cross-Correlation loss (NCC) of intensity images were used as the first step, following a second registration using SSIM loss and NCC of intensity images. The optimal IoU threshold for the association algorithms is 0.1.
50 integral glomerular identifications were randomly selected for volume estimation. A semi-manual annotation tool EasierPath13 is used to generate ground truth.
3.3. Evaluation Metrics
For the purpose of tracking results, the standard MOT metrics from MOT-Challenge 201514 were used for the purpose of verification. All human annotation and automatic tracking results were saved in the MOT-Challenge 2015 format to be compatible with the official evaluation code. IDF1, IDP, and IDR (the larger, the better) were the ratio of correctly identified detections over the number of ground-truth and computed detections according to the F1, recall, and precision scores. MOTA, MOTP, and MOTAL (the larger, the better) were the multiple objects tracking accuracies, which combine false positives, missed targets, and identity switches metrics.
For voxel estimation evaluation, standard statistical measures including means, medians, standard deviations (SD), ranges, and other distribution measures were used to assess the accuracy of the three estimation methods with human annotation. The Spearman correlation coefficient is deployed to present relationships between each estimation method and human annotation. The Bland-Altman plot is implemented to display the limits of agreement for each estimation method with human annotation.
4. RESULTS
Figure 3 presents the qualitative performance of Map3D+ on MOT. It shows that our method can eliminate false-positive objects, while the disconnections from insufficient global affine registration are resolved. Table 1 presents the quantitative performance of Map3D+ compared with our previous study. Our latest method can achieve better performances in most of the MOT evaluation metrics.
Figure 3.
This figure demonstrates the MOT results by using Map3D+. The false-positive objects can be diminished by segmentation. Multi-modality registration can assist in reconnecting discontinued identifications.
Table 1.
MOT performance on CircleNet detection results.
Figure 4 displays the Spearman coefficients with different estimation approaches compared with manual annotation. Bland-Altman plots are also provided to show the differences in each glomerulus with different methods. 70 μm is approximately the diameter of a mouse glomerulus, which determines the approximately glomerular volume is 0.2 ×10−3mm3. All the methods are estimated over-size of glomerular because the segmentation fails to cut off multiple glomerular areas when they are close. Another factor that might magnify the volume calculation is that the training data includes atubular glomeruli, whose annotations are larger than connected glomeruli. Although there is a systematic overestimation of glomerular sizes, our method achieves consistent estimation performance with a higher correlation score and smaller differences.
Figure 4.
This figure shows the relationships between annotations and volume estimations. The first row presents Spearman coefficients among different volume estimations. The second row presents the Bland-Altman plot among different volume estimations. The red circles are outliers, which significantly contribute to errors.
5. DISCUSSION
In this study, we introduce an end-to-end holistic deep-learning-based method that achieves semantic segmentation, patch-wise registration, and association algorithm for each glomerulus present in WSI. This algorithm achieves automatic detection, segmentation and multi-object tracking (MOT) of individual glomeruli with large-scale glomerular-registered assessment in a 3D context. Our study provides an approach for extensive 3D glomerular reconstruction and volume estimation from serial section WSIs. The future technical improvements would be: (1) to design a global deformation on WSIs with a deep-learning-based method to replace patch-wise registration with better registration performance and less time cost, (2) to design a better segmentation model for glomeruli to segregate adjacent glomeruli, and (3) to design a neat visualization optimization for 3D phenotype analysis with a precise voxel estimation.
ACKNOWLEDGMENTS
This work was supported in part by NIH NIDDK DK56942(ABF). This work was supported in part by a grant from the Skin Cancer Foundation and the Dermatology Foundation. This work has not been submitted for publication or presentation elsewhere.
Footnotes
Kidney Int. 2017 Dec;92(6):1395–1403.
Kidney Int. 2020 Nov 1;S0085-2538(20)31240-0.
REFERENCES
- [1].Aeffiner F, Zarella MD, Buchbinder N, Bui MM, Goodman MR, Hartman DJ, Lujan GM, Molani MA, Parwani AV, Lillard K, et al. , “Introduction to digital image analysis in whole-slide imaging: a white paper from the digital pathology association,” Journal of pathology informatics 10 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- [2].Huo Y, Deng R, Liu Q, Fogo AB, and Yang H, “Ai applications in renal pathology,” Kidney international (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- [3].Rehem JMC, dos Santos WLC, Duarte AA, de Oliveira LR, and Angelo MF, “Automatic glomerulus detection in renal histological images,” in [Medical Imaging 2021: Digital Pathology], 11603, 116030K, International Society for Optics and Photonics (2021). [Google Scholar]
- [4].Jayapandian CP, Chen Y, Janowczyk AR, Palmer MB, Cassol CA, Sekulic M, Hodgin JB, Zee J, Hewitt SM, O’Toole J, et al. , “Development and evaluation of deep learning–based segmentation of histologic structures in the kidney cortex with multiple histologic stains,” Kidney international 99(1), 86–101 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- [5].Bouteldja N, Klinkhammer BM, Bülow RD, Droste P, Otten SW, von Stillfried SF, Moellmann J, Sheehan SM, Korstanje R, Menzel S, et al. , “Deep learning–based segmentation and quantification in experimental kidney histopathology,” Journal of the American Society of Nephrology 32(1), 52–68 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- [6].Ginley B, Lutnick B, Jen K-Y, Fogo AB, Jain S, Rosenberg A, Walavalkar V, Wilding G, Tomaszewski JE, Yacoub R, et al. , “Computational segmentation and classification of diabetic glomerulosclerosis,” Journal of the American Society of Nephrology 30(10), 1953–1967 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- [7].Deng R, Yang H, Jha A, Lu Y, Chu P, Fogo AB, and Huo Y, “Map3d: registration based multi-object tracking on 3d serial whole slide images,” IEEE Transactions on Medical Imaging (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- [8].Yang H, Deng R, Lu Y, Zhu Z, Chen Y, Roland JT, Lu L, Landman BA, Fogo AB, and Huo Y, “Circlenet: anchor-free glomerulus detection with circle representation,” in [International Conference on Medical Image Computing and Computer-Assisted Intervention], 35–44, Springer; (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
- [9].Sarlin P-E, DeTone D, Malisiewicz T, and Rabinovich A, “Superglue: Learning feature matching with graph neural networks,” in [Proceedings of the IEEE/CVF conference on computer vision and pattern recognition], 4938–4947 (2020). [Google Scholar]
- [10].Chen L-C, Papandreou G, Schroff F, and Adam H, “Rethinking atrous convolution for semantic image segmentation,” arXiv preprint arXiv:1706.05587 (2017). [Google Scholar]
- [11].Lee MC, Oktay O, Schuh A, Schaap M, and Glocker B, “Image-and-spatial transformer networks for structure-guided image registration,” in [International Conference on Medical Image Computing and Computer-Assisted Intervention], 337–345, Springer; (2019). [Google Scholar]
- [12].Lane PH, Steffes MW, and Mauer SM, “Estimation of glomerular volume: a comparison of four methods,” Kidney international 41(4), 1085–1089 (1992). [DOI] [PubMed] [Google Scholar]
- [13].Zhu Z, Lu Y, Deng R, Yang H, Fogo AB, and Huo Y, “Easierpath: an open-source tool for human-in-the-loop deep learning of renal pathology,” in [Interpretable and Annotation-Efficient Learning for Medical Image Computing], 214–222, Springer; (2020). [Google Scholar]
- [14].Leal-Taixé L, Milan A, Reid I, Roth S, and Schindler K, “MOTChallenge 2015: Towards a benchmark for multi-target tracking,” arXiv:1504.01942 [cs] (Apr. 2015). arXiv: 1504.01942. [Google Scholar]