Feasibility of artificial intelligence-based decision supporting system in tolvaptan prescription for autosomal dominant polycystic kidney disease

Abstract

Purpose

Total kidney volume (TKV) measurement is crucial for selecting treatment candidates in autosomal dominant polycystic kidney disease (ADPKD). We developed and investigated the performance of fully-automated 3D-volumetry model and applied it to software as a service (SaaS) for clinical support on tolvaptan prescription in ADPKD patients.

Materials and Methods

Computed tomography scans of ADPKD patients taken between January 2000 and June 2022 were acquired from seven institutions. The quality of the images was manually reviewed in advance. The acquired dataset was split into training, validation, and test datasets at a ratio of 8.5:1:0.5. Convolutional, neural network-based automatic segmentation model was trained to obtain 3D segment mask for TKV measurement. The algorithm consisted of three steps: data preprocessing, ADPKD area extraction, and post-processing. After performance validation with the Dice score, 3D-volumetry model was applied to SaaS which is based on Mayo imaging classification for ADPKD.

Results

A total of 753 cases with 95,117 slices were included. The differences between the ground-truth ADPKD kidney mask and the predicted ADPKD kidney mask were negligible, with intersection over union >0.95. The post-process filter successfully removed false alarms. The test-set performance was homogeneously equal and the Dice score of the model was 0.971; after post-processing, it improved to 0.979. The SaaS calculated TKV from uploaded Digital Imaging and Communications in Medicine images and classified patients according to height-adjusted TKV for age.

Conclusions

Our artificial intelligence-3D volumetry model exhibited effective, feasible, and non-inferior performance compared with that of human experts and successfully predicted the rapid ADPKD progressor.

Graphical Abstract

INTRODUCTION

Autosomal dominant polycystic kidney disease (ADPKD) is the most common hereditary and progressive disease with an average prevalence of 2.7 in 10,000 people. ADPKD accounts for approximately 10% of end-stage renal disease and is characterized by progressive, fluid-filled cyst formation originating from 1% to 2% of the nephrons [1]. The kidney gradually enlarges due to compensatory hyperfiltration in non-cystic nephrons and this renal hypertrophy results in pain, a palpable mass, hypertension, hematuria, cyst rupture, cyst infection, and ultimately, renal failure [2]. There has been no definite treatment for ADPKD until the 2010s, although tolvaptan, a vasopressin V2 receptor antagonist, has shown efficacy in reducing the annual increase in total kidney volume (TKV) and renal function deterioration [3]. TKV is an important parameter for predicting disease progression and has been suggested as a key prognostic factor to identify patients who are at risk of rapid progression to renal failure [4,5].

Traditionally, manual labeling of kidney contours on each computed tomography (CT) scan or magnetic resonance image (MRI) has been considered the most accurate TKV measurement system [6,7]. However, the entire process is laborious, time-consuming, and carries the risk of human error and interobserver discrepancy. Consequently, robust investigations of more efficacious and accurate tools for TKV calculation have been conducted. Numerous studies have reported the functional outcomes of semi-automated measurement and deep learning-based artificial intelligence (AI) volumetry in various image modalities, including CT, MRI, and ultrasound (US) [8]. We had previously developed an AI-based volumetry framework for polycystic kidney and liver diseases [9]. The performance of our AI was fast and accurate with outcomes that are highly comparable to those from human specialists. In the present study, we upgraded, trained, and applied our deep learning volumetry system to larger clinical datasets and investigated its performance on ADPKD patients. Next, we applied our AI-based volumetry to develop software as a service (SaaS) which is based on Mayo imaging classification to identify the ADPKD rapid progressors.

MATERIALS AND METHODS

1. Ethics and data availability statement

This study was approved by the Institutional Review Board (no. 2022-08-028) of the Ewha Womans University Mokdong Hospital, and informed consent was waived due to the retrospective study design. In addition, current study was performed according to TRIPOD-AI reporting guideline. The reference source cord for GitHub is follows: https://github.com/Project-MONAI/tutorials/tree/main/modules/dynunet_pipeline.

2. Dataset information and study population

We have established a polycystic kidney CT database, which includes both raw data and a hand-labeled, semantic-segmentation dataset inspected by experts [9]. ADPKD-related data were sorted from the existing raw data and additional CT images of ADPKD cases taken between January 2000 and June 2022 were acquired from seven hospitals under two medical centers. Patients undergoing dialysis at the time of the CT scan and those with any noisy data on CT scans were excluded. All images were manually reviewed by radiologists in advance. Pooled data from institutions were combined to construct a CT-ADPKD database.

3. Overview of the training process

The TKV calculation algorithm of our study extracts the ADPKD area by first analyzing the abdominopelvic CT images based on the well-trained, semantic-segmentation model named “3D Dynamic U-Net”, a type of convolutional neural network-based segmentation model. The AI segmentation model first undergoes a process of learning large amounts of data to transform and distinguish functional relationships of data consisting of pairs of a 3D CT array and a correct-3D CT Mask array. The overall workflow of the algorithm consists of three steps: data preprocessing, ADPKD area extraction, and post-processing. After going through all the processes of the algorithm, a 3D segment mask is obtained and TKV is calculated.

4. Preprocessing

The range of Hounsfield Unit values considered meaningful were clipped and converted to a float32 data type in the range of 0.0 to 1.0 so that the graphic process unit could accelerate it. Next, missing values were input, noisy data were smoothed, identified outliers were removed, and solvent data inconsistency was checked. The process of handling noisy data was replaced with an image preview by radiologists. In the case of patients with duplicated process IDs, only samples with non-overlapping series were included in the dataset.

5. ADPKD area extraction

We applied a network based on a 3D Dynamic U-Net for ADPKD segmentation. The model was provided by the MONAI Project, which is an open-source project for the latest model baseline codes associated with medical AI (e.g., 3D Dynamic U-Net, 2D/3D Residual U-Net, and 3D UNetR). Either 2D or 3D arrays were input to 3D Dynamic U-Net, and residual block and deep supervision were applied for learning. Deep supervision calculates the final loss by multiplying each loss term by weight after calculating the loss between all featured maps and the ground-truth mask images (Fig. 1).

Fig. 1. Dynamic 3D U-net architecture. The model architecture used for 3D segmentation mask inference for autosomal dominant polycystic kidney disease total kidney volume calculation. Conv, convolutional; GT, ground-truth mask.

In addition, instance normalization was robustly applied to learn the heavy model, even in small batch sizes. To prevent 3D input data from causing out-of-memory problems, we split the input data into small 3D patches. In this process, 3D patches were extracted in a sliding window to prevent the occurrence of grid pattern artifacts in the segmentation mask inferred by the model. The batch size was set to 1 (one case), and the learning rate was set to decay according to (1-epoch/max_epochs)×0.9, and training was performed for 220 epochs. Data augmentation was applied with the following settings:

• - Random zoom: min_zoom 0.9, max_zoom 1.2 (15%)

• - Random Gaussian noise: std 0.01 (15%)

• - Random vertical flip (50%)

6. Post-processing

Area extractions performed by AI sometimes contain false-alarm cases with holes between slices. Although these false-alarm cases might not be a major problem in the TKV calculation, they should be removed during post-processing as they negatively impact the overall quality of the 3D reconstruction. As the kidney should maintain a constant shape without breaking, the slice with a false alarm was interpolated and filled. The pseudo-code of the false-alarm removal algorithm was executed on both kidneys, and images were merged to obtain a 3D mask of the kidney with complete connectivity.

7. TKV calculation

The 3D mask volume was constructed using the inferred segmentation masks after completion of all the mentioned processes. TKV was calculated by applying the 3D volume to the volume calculation equation:

TKV (mm³)=pixel spacing (width, height)×slice thickness (depth)×total number of voxel

The full flow chart of our algorithm is graphically summarized in Fig. 2.

Fig. 2. Overall flow chart of the algorithm. (A) Abdomen and pelvis computed tomography (CT) images were acquired from several hospitals. (B) Dataset was annotated with hand-labelled segmentation mask. (C) 3D Dynamic U-net model was trained with the dataset, which was previously constructed with 38,261 hand-labeled semantic-segmentation and inspected by three expert radiologists. (D) Automatic segmentation was performed to test autosomal dominant polycystic kidney disease samples with the trained model. (E) Collected masks inferred by the model underwent post-processing to remove false alarms. (F) 3D volume model was constructed with inferred segmentation masks. (G) Total kidney volume (TKV) calculation was performed by applying the 3D volume to the volume calculation equation.

8. Development of SaaS from AI-based kidney volumetry

After validation of our AI-based kidney volumetry, we developed SaaS (SYN-IGN-AN, Ignite™, Synergy A.I. Co. Ltd.) to identify the rapid progressors among ADPKD patients. Mayo imaging classification is currently the best prediction model for selecting rapid progressors among ADPKD patients and is also a good predictor for Korean ADPKD population [10]. The SaaS included uploading of Digital Imaging and Communications in Medicine (DICOM) images for TKV measurement, and other variables including age, sex, race, height, serum creatinine, and genotype.

9. Statistical analysis

The acquired dataset was split into training, validation, and test data at a ratio of 8.5:1:0.5 to investigate the performance of the model. The pixel array shape of each DICOM file was 512×512×S(n). To calibrate the discrepancy in the number of slices for each sample, padding filled with zeros was added to the cases with less than 250 slices.

To evaluate the performance of the medical image analysis algorithm, the dice similarity correlation (DSC) score was used as a statistical validation metric to assess the accuracy of spatial overlap between the true and predicted manual segmentation labels using the following formula:

DSC=(2×True Positive)/(True Negative+False Positive+True Positive+False Negative)

Before calculation, the 3D masks of the kidney region were obtained from 32 test cases using a well-trained model (the number of slices may be different in each case). After the model inferred the 3D masks for 32 test samples, a confusion matrix was constructed and the DSC calculated. Five-fold cross-validation was performed to confirm the robustness of the model for the training dataset. The comparison between ground-truth and auto-segmented TKV was also performed with calculation of an interclass correlation coefficient (ICC).

RESULTS

1. Dataset information

A total of 753 cases with 95,117 slices were included in the dataset. CT images were acquired from eight types of CT scanners in seven institutions. The average number of slices in the entire dataset was 126 (range 26–276). The model was trained with 655 CT-DICOM files with 82,850 slices, validated with 66 DICOM files with 7,778 slices, and tested with 32 DICOM files with 4,489 slices. The detailed information of our database is summarized in Table 1.

Table 1. Baseline information of participating hospitals, types of CT images, and the model names of the CT scanners.

		Hospitals							Total
		Hallym-Chuncheon	Hallym-Pyeongchon	Hallym-Kangdong	Hallym-Kangnam	Hallym-Dongtan	Ewha-Mokdong	Ewha-Seoul	Total DICOM
Dataset (n)
	Noncontrast	27	54	41	37	23	156	128	753	95,117
	Contrast	6	9	3	4	10	138	59
Model name of the CT scanners		SOMATOM Definition Flash (Siemens)	SOMATOM Sensation 64 (Siemens) SOMATOM Definition Edge (Siemens) SOMATOM Definition Flash (Siemens)	OMATOM Smile (Siemens) MX8000 IDT (Philips)	Brilliance CT 64-Channel (Philips) MX8000 IDT (Philips) SOMATOM Sensation 64 (Siemens)	Brilliance Big Bore (Philips)	SOMATOM Sensation 64 (Siemens)	SOMATOM Force (Siemens)	8 CT scanners 7 Institutions

CT, computed tomography; DICOM, Digital Imaging and Communications in Medicine.

2. Performance of the algorithm

The differences between the ground-truth ADPKD kidney mask and the predicted ADPKD kidney were negligible (Fig. 3). The ICC calculated by comparing the ground truth and AI auto-segmentation TKV results was 0.99976, which is extremely reliable (Fig. 4). Most of the differences were false alarms, which were successfully removed by the post-process filter; as shown in Fig. 5, they had no clinical significance. The test-set performance plot shows homogeneously equal performance in each of the 32 test samples (Fig. 6).

Fig. 3. Comparison between ground truth and artificial intelligence auto-segmentation. (A) Original computed tomography slice image. (B) Ground-truth mask image superimposed onto the original input image. (C) Predicted mask image superimposed onto the original image. (D) The differences between (B) and (C) relative to the original images. IoU, intersection over union.

Fig. 4. The interclass correlation coefficient (ICC) between ground truth and artificial intelligence (AI) auto-segmentation.

Fig. 5. Representative image of post-processing filter. (A) Before post-process filter application. (B) After post-process filter application. (C) Comparison of rendering 3D space before and after post-process filter.

Fig. 6. Test set performance plot. The solid red line represents the dice similarity correlation (DSC) score of the background class for each test sample, and the solid black line represents the DSC score of the autosomal dominant polycystic kidney disease (ADPKD) kidney class for each test sample. The red and the black dotted lines denote the average DSC scores of the background class and the ADPKD kidney, respectively.

The test DSC score of our model was 0.971 and the DSC score improved to 0.979 after post-processing (Table 2).

Table 2. Performance of the artificial intelligence auto-segmentation model.

Model	Test Dice score	Validation Dice score
3D-Dynamic U-Net	0.971	0.967
After post-processing (for 3D model)	0.979	0.970

3. Operation of AI-based kidney volumetry

After dragging-and drop of DICOM files from the CT scan, patients’ baseline demographics including age, sex, race, height, and serum creatinine is entered (Fig. 7A). After inference of the data, kidney volume is calculated automatically and ADPKD classification is reported (Fig. 7B). The overall process for using our SaaS is presented in Supplementary Video.

Fig. 7. Representative process for utilizing Ignites. (A) Default page for patient information input. Digital Imaging and Communications in Medicine (DICOM) files for kidney volume calculation can be uploaded. Patient demographics including age, weight, height, serum creatinine, and autosomal dominant polycystic kidney disease (ADPKD) genotypes are entered for ADPKD classification. (B) ADPKD classification of the selected patient.

DISCUSSION

The Mayo imaging classification is a simple tool that can easily identify patients at the highest risk for disease progression regardless of renal function and is currently used for tolvaptan treatment candidate selection [11,12]. Accurate TKV measurement is crucial for prompt decision-making; however, TKV is generally approximated with ellipsoidal kidney length, width, and depth measurements due to the lack of feasible, commercialized, technical support. The present study focus on kidney volume only and presents the excellent performance of a well-trained, deep learning volumetry system, a 3D-Dynamic U-Net, in the largest clinical dataset, which can be helpful in overcoming the aforementioned drawbacks. Our AI model was fully automated and had a Dice score of 0.971, which was further improved to 0.979 by removing clinically insignificant data via post-processing. Moreover, our AI model was successfully applied to SaaS which is designed to identify rapid progressors.

Current studies on deep learning-based volume measurement in ADPKD utilize various image modalities, including CT scans, MRI, and US. The Dice score of each segmentation methodology ranges from 0.86 to 0.969 for CT scans [9,13,14] and 0.84 to 0.98 for MRIs [15,16,17,18,19]. There are few studies on US; one study by Jagtap et al. [20] reported a dice score of 0.80 by 2D U-Net segmentation in kidney US. Our 3D-Dynamic U-Net provided the highest Dice score of 0.971 compared to previous studies using CT scans. This excellent, expert-level performance is based on a large clinical dataset. Our dataset included 95,117 slices from 753 ADPKD patients, which is the largest number of study subjects reported so far.

Early deep learning-based methods usually consist of semi-automated systems in which AIs produce the initial prediction of renal contours that can later be confirmed and modified by radiologists [21,22]. Such a system omits the process of laborious manual drawing on each slide, resulting in time-cost effectiveness, efficiency, accuracy, and improved ergonomics for the physicians. However, a semi-automated system always requires expert manpower for supervision, and there is always a risk for human error. Volumetry, including TKV measurement, is a field of work that requires consistency and reproducibility. This has resulted in the need for semi-automated segmentation in order to overcome the limitations of manual annotation. Our AI model provides a fully automated segmentation technique with a Dice score of 0.979, which is non-inferior to human specialists who had DSC ranging from 0.957 to 0.975 based on our previous report [9].

Among various segmentation models, we chose 3D Dynamic U-Net as the main network of the present study rather than 2D/3D Residual U-Net, 3D-UnNetR, and the 3D-Dynamic U-Net 5-Fold ensemble set. Three-dimensional Dynamic U-Net is implemented based on the nnU-Net architecture proposed by Isensee et al. [23] and has three advantages compared to the general vanilla U-Net. First, residual connection is supported in convolutional blocks. Second, anisotropic kernel sizes and strides can be used in each layer. Third, deep supervision heads can be added. Moreover, 3D Dynamic U-Net generates full-resolution feature maps at multiple semantic levels owing to the nested skip pathways. Thus, the losses are estimated from four semantic levels. The deep supervision enables model pruning and improves, or in the worst case, achieves comparable performance to using only one loss layer.

Our segmentation model includes an important refinement process named “post-processing”. Post-processing identifies and removes false alarms, which usually have no clinical significance, leading to an improvement of the Dice score in both validation and test sets. The differences between pre- and post-process filters might not be significant (0.971 vs. 0.979) as the performance of 3D-Dynamic U-Net itself is highly competitive. However, such processes should not be neglected as aesthetic mask quality improvement provides more accurate and reliable 3D images for both clinicians and patients. As a 3D printed model is commonly used to facilitate more intuitive explanation to patients, it is crucial that the quality of 3D images be reliable. This enables better understanding of patients regarding their clinical information and might lead to higher compliance and persistence rate to the clinicians.

Five years have passed since the approval of tolvaptan for kidney disease, but several problems remain unsolved in clinical practice. Efficacy monitoring of any newly adopted drug is necessary for objective and scientific evaluation. The therapeutic efficacy of tolvaptan is frequently monitored by changes in serum glomerular filtration rate (GFR) but not consecutive TKV measurements due to the absence of reliable methodologies in clinical practice [4].

Consecutive TKV measurements are important and helpful for treatment efficacy monitoring. First, there are other comorbidities, including uncontrolled hypertension, diabetes, and vascular disease, which could be modifiable contributing factors for GFR decline in ADPKD patients [24,25]. Serial TKV monitoring is necessary to confirm that renal function deterioration is actually due to ADPKD progression. Second, there is still no definite titration regimen for tolvaptan associated with maximal therapeutic efficacy and minimal adverse events [4,26]. The combination of GFR and TKV on serial follow-ups might be useful to precisely determine the effects of tolvaptan persistence on the clinical course of ADPKD. This, in turn, could lead to the establishment of a clear treatment strategy. Third, tolvaptan is currently the only drug being widely prescribed for ADPKD and is extremely expensive. Patients need to take the medication twice a day until they progress to end-stage renal disease. The duration of treatment could be a lifetime if the side effects [27,28] (especially liver toxicity and polyuria) are tolerable and renal function is maintained. This is a serious socio-economic burden for both the affected individuals and society. Hence, it is very important to provide scientific evidence to decide which types of patients should get medical expense coverage for tolvaptan usage.

Despite the need for sufficient research on TKV measurement, there are few 3D volumetry techniques capable of effective and accurate calculation. Variations in image modalities, models, protocols, and the performance of radiologists were considered insurmountable in traditional settings. Moreover, valid and sensitive approaches for monitoring the changes in TKV have not yet been established. Our AI 3D-volumetry could be a breakthrough for such problems as it is capable of high-quality performance in datasets where most images are non-enhanced CT scans, which do not require any special protocol (although radiation exposure dose and image thickness might differ).

The limitations of the present study are as follows. We needed a training dataset created by experts to set ground-truth images and this dataset only included CT scans. However, there is no other way for setting ground-truth images except for hand-labeled semantic segmentation. In addition, CT scan is more favorable than MRI as setting a ground-truth value in low-resolution MRI is extremely difficult to be generalized based on differences in vendor and hospital setting values. Our AI volumetry presented high-quality performance despite the use of non-enhanced CT scans.

We report an automatic 3D-volumetry model from various CT models by using the largest clinical dataset reported to date; this successfully reflects the heterogeneity of real-world settings. Furthermore, post-processing of the predicted 3D images improved image quality with better aesthetics, accuracy, and reliability. Furthermore, we developed SaaS based on Mayo imaging classification to support clinical decision on tolvaptan prescription.

Tolvaptan is the only Food and Drug Administration-approved drug for ADPKD treatment. However, currently available studies are mostly clinical trials on ADPKD patients who were mostly recruited from Western countries; therefore, robust investigations on other ethnicities and studies that reflect real-world data are urgently required. Above all, future studies should utilize more comprehensive clinical data by including not only serum GFR but also TKV. If our AI-based segmentation model is more widely applied to real practice, non-inferior and high-quality data could be provided to both physicians and patients without laborious and time-consuming, manual labeling by radiologists. In addition, further clinical decision-making, including therapeutic outcome monitoring, and prediction of prognosis, might be more efficacious, leading to decreased medical expenses. Future studies with external validation and comparison with other segmentation models are warranted for the clinical application of this outstanding AI volumetry technique in real-world settings.

CONCLUSIONS

Our deep learning-based, automatic-segmentation, 3D volumetry exhibited effective, feasible, and non-inferior performance compared with that of human experts. This model was improved by post-processing to provide more accurate and reliable 3D images for both clinicians and patients. Moreover, application of AI 3D-volumetry to SaaS for selecting rapid progressors was also successful.

SUPPLEMENTARY MATERIAL

Accompanying video can be found in the ‘Urology in Motion’ section of the journal homepage (www.icurology.org). The supplementary video clip can also be accessed by scanning a QR code, and will be available on YouTube: Supplementary Video, https://youtu.be/gVh72uY2cSU.