Accuracy and efficiency of an artificial intelligence-based pulmonary broncho-vascular three-dimensional reconstruction system supporting thoracic surgery: Retrospective and prospective validation study

Summary

Background

Anthropomorphic phantoms are used in surgical planning and intervention. Ideal accuracy and high efficiency are prerequisites for its clinical application. We aimed to develop a fully automated artificial intelligence-based three-dimensional (3D) reconstruction system (AI system) to assist thoracic surgery and to determine its accuracy, efficiency, and safety for clinical use.

Methods

This AI system was developed based on a 3D convolutional neural network (CNN) and optimized by gradient descent after training with 500 cases, achieving a Dice coefficient of 89.2%. Accuracy was verified by comparing virtual structures predicted by the AI system with anatomical structures of patients in retrospective (n = 113) and prospective cohorts (n = 139) who underwent lobectomy or segmentectomy at the Peking University Cancer Hospital. Operation time and blood loss were compared between the retrospective cohort (without AI assistance) and prospective cohort (with AI assistance) for safety evaluation. The time consumption for reconstruction and the quality score were compared between the AI system and manual reconstruction software (Mimics®) for efficiency validation. This study was registered at https://www.chictr.org.cn as ChiCTR2100050985.

Findings

The AI system reconstructed 13,608 pulmonary segmental branches from retrospective and prospective cohorts, and 1573 branches of interest corresponding to phantoms were detectable during the operation for verification, achieving 100% and 97% accuracy for segmental bronchi, 97.2% and 99.1% for segmental arteries, and 93.2% and 98.8% for segmental veins, respectively. With the assistance of the AI system, the operation time was shortened by 24.5 min for lobectomy (p < 0.001) and 20 min for segmentectomy (p = 0.007). Compared to Mimics®, the AI system reduced the model reconstruction time by 14.2 min (p < 0.001), and it also outperformed Mimics® in model quality scores (p < 0.001).

Interpretation

The AI system can accurately predict thoracic anatomical structures with higher efficiency than manual reconstruction software. Constant optimization and larger population validation are required.

Funding

This study was funded by the Beijing Natural Science Foundation (No. L222020) and other sources.

Research in context.

Evidence before this study

A literature search of all published studies was performed using PubMed and EMBASE from inception to November 2021 to identify studies using “artificial intelligence (AI)”, “deep-learning”, “three-dimensional reconstruction”, “surgical anatomy” and “accuracy or efficiency”. No restrictions on study type or language were applied. Anthropomorphic phantoms are swiftly used in surgery, and ideal accuracy and high efficiency are prerequisites for their clinical application. The AI system has been postulated to be highly efficient in reconstruction due to its fully automated properties to reduce the workload of surgeons. New phantom systems based on AI technique disclose distinguished potency in supporting surgical planning and navigation. Previous studies predominantly investigated the Dice coefficient, evaluating the segmentation performance of medical images by calculating the statistical validation of manual annotation in CT images, but validation in real surgical scenarios is lacking. To our knowledge, no other studies have verified the accuracy of automated 3D reconstruction models using anatomical structures as the standard in a prospective cohort with strict statistical design.

Added value of this study

This study aimed to propose an AI-based 3D reconstruction system for supporting thoracic surgery, which was optimized by multitask learning, attention mechanisms, and deep supervision. This AI system has high modularity, in which the modules are designed using specific segmentation algorithms for different pulmonary tissues, and each module can run and be optimized independently. A reasonably stable level of the Dice similarity coefficient reached 89.2% after 500 training cases. To further validate the accuracy of the 3D model in clinical utilization, we compared the virtual 3D broncho-vascular structure phantoms with anatomical structures in a retrospective cohort and a prospective cohort, achieving 100% and 97% accuracy for segmental bronchi, 97.2% and 99.1% for segmental arteries, and 93.2% and 98.8% for segmental veins, respectively. Meanwhile, operation time and intraoperative blood loss were compared between the retrospective cohort (without AI assistance) and prospective cohort (with AI assistance) for safety verification. With the assistance of the AI system, a 3D model prior to surgery could help to reduce operation time while causing imperceptible differences in intraoperative blood loss. Its efficiency was also verified by comparison with a manual reconstruction system, and the AI system required significantly less time for 3D model reconstruction procedures and had higher visual quality scores for 3D model assessment.

Implications of all the available evidence

With the development of computer technology and AI, anthropomorphic phantoms, such as 3D reconstruction, may help to enhance surgeons’ steric perception, simplify the training process and lower surgical morbidity by preoperatively predicting anatomical structures. This study demonstrates that phantom 3D models supporting thoracic procedures developed either by AI or other manual labeling systems can achieve high accuracy in anatomical structure prediction, but the AI system shows higher efficiency in terms of reconstruction time and visual performance. The operation time may also be shortened with the assistance of 3D reconstruction model.

Introduction

An anthropomorphic 3D model is a fundamental tool for preoperative planning and intraoperative navigation in clinical practice. Interactive 3D models have been proven to significantly enhance surgeons’ steric perception and lower surgical morbidity by preoperatively predicting anatomical structures.^1,2 Previously, 3D models accurately predicted the liver resection volume and margin in hepatic surgery³ and overwhelmingly reduced the abnormal rate of femoral anteversion restoration in hip arthroplasty.⁴

In the field of thoracic service, anatomic lung resection is an oncological curative procedure for early-stage lung cancers.⁵ Pulmonary vessel variation occurred in 20–30% of patients,⁶ and 78.5% of thoracic surgeries with intraoperative vascular injuries are converted to open procedures in the era where VATS is readily accessible.⁷ Knowledge of the branching pattern of pulmonary vessels was proved essential for preoperative planning of complete VATS.⁸ Previously, senior surgeons would always mentally reconstruct what they watched on 2D pictures into a 3D architecture where all details are likely to be represented as they would appear intraoperatively.⁹ With the development of AI technology, 3D models can simplify reconstruction procedures and assist surgeons in a less technical-requesting fashion. Clinical study has demonstrated that preoperative anatomical 3D reconstruction models could offer comprehensive anatomy to facilitate complete VATS surgery safely.¹⁰

Pulmonary 3D reconstruction models can be completed using semi-automation tools (such as Mimics®, OsiriX, and 3D slicer), which can simulate anatomic structures, clarify the division of pulmonary segments, and determine the location of the lesion.¹¹ However, professional skill requirements, manual annotation, and high time consumption potentially curb the enthusiasm for widespread application of these systems in clinical practice.¹² AI techniques may automatically learn from raw data and rapidly complete the 3D model to improve clinical efficiency.¹³ Since preoperative 3D reconstruction is increasingly applied in different surgical fields, validation of its accuracy and safety in a large cohort of patients is urgently needed to support its clinical application.

We established a fully automated reconstruction system based on deep learning using a 3D convolutional neural network (CNN) and improved its performance by multi-task learning, attention mechanism, and deep supervision. A reasonably stable level of the Dice similarity coefficient reached 89.2% after 500 training cases. Second, we compared the virtual broncho-vascular structure phantom of the AI system (AI-based 3D reconstruction system) with anatomical structures to demonstrate its accuracy in a retrospective cohort and a prospective cohort. Meanwhile, operation time and intraoperative blood loss were compared between the retrospective cohort (without AI assistance) and prospective cohort (with AI assistance) for safety verification. Third, we compared the AI system with a manual reconstruction system in terms of accuracy, quality score, and total reconstruction time for efficiency verification.

Methods

Introduction of AI system

System principle and composition

The AI-based 3D reconstruction system was designed to assist preoperative thoracic surgery design based on deep learning, using an “end-to-end” learning process to extract features automatically from training data for various tasks, such as segmentation, classification, and detection. A CNN (referring to 3D V-Net) was used for the segmentation of lobes, bronchi, and blood vessels through an attention mechanism, and then 3D reconstruction models were completed and smoothed using the marching cube algorithm.

The architecture of the AI system consisted of common pre-processing, segmentation, and post-processing modules (Fig. 1), and normalized CT scans (window level, −300 HU; window width, 1800 HU) were used in lung segmentation to reduce extrapulmonary noise. During the training process, a 3D patch size of 64 × 196 × 196 was selected for pulmonary bronchi, arteries, and vein segmentation owing to the GPU memory bottleneck. The V-Net was designed for segmentation using 3D contour variance information, and the corresponding loss function was designed for a clearer contour. The attention module named ‘Project & Excite’ and the distance map as prior knowledge were embedded into V-Net to optimize the network performance for bronchi and blood vessel segmentation. Nodules were detected using a 3D faster region-based CNN (Faster R-CNN) network of the deep learning model, and their regions were segmented using a 2D U-Net network based on the detected bounding boxes. In the post-processing module, the segmentation results are processed before being rendered by traditional techniques, such as noise reduction, while maintaining the largest connected component.

Fig. 1.

Overview of AI system. AI system consisted of three main parts: pre-processing, model training and postprocessing. To improve the performance and robustness of reconstruction, we used lung segmentation, data augmentation and random cropping in pre-processing before the training phase. During the training process, a 3D patch size of 64 × 196 × 196 was selected for pulmonary bronchus, artery, and vein segmentation due to the GPU memory bottleneck, and segmentation of lobes and nodules was completed by V-Net and Faster-RCNN. Segmentation models were processed by computer graphical operations before surface rendering.

Optimization of AI systems

According to the basic theory of deep learning, neural networks would achieve better performance with more high-quality data, which would lead to stable convergence of the neural network in the training phase based on gradient descent. To elucidate the optimization of the AI system, we chose one patient with a nodule in the left upper lobe, whose CT scan was reconstructed for a 3D model by the AI system in different training stages with 10, 30, 60, 100, 200, 300, 400, and 500 cases. The Dice similarity coefficient, which refers to the average concordance rate for pixel identification between AI systems and humans, was treated as a metric in image segmentation. The Dice coefficient index of our AI system showed a logarithmic growth with an increasing number of training data improved from 62.8% to 89.2%, and the manual modification time decreased from 68 min to 0 min correspondingly (Fig. 2). The manual modification was completed by senior surgeons and senior AI algorithm engineers who have been well-trained in annotating CT imaging with extensive experience. They worked together to modify the 3D models for the clear exhibition of the anatomical segmental branches to meet the clinical demands using high-resolution CT scanning reviewed by experienced surgeons as the standard. Then the corrected 3D models were learned by the AI system as feedback information, by which the AI system can gradually complete the 3D models accurately. When enough cases were included in the training dataset, AI-based 3D models could be well improved to meet the clinical demands, and no modification was needed. As the AI system was trained with 10 cases, misidentification of the 3D reconstruction model occurred at the vascular trunk and arteriovenous boundaries (Fig. 3a). When trained with 60 cases, vascular trunks of the 3D model were identified, whereas arteriovenous boundaries were misidentified (Fig. 3b). Only small blood vessels were misidentified when the AI system was trained with 100 cases compared to the standard reference (Fig. 3c). The 3D reconstruction model of the AI system trained with 500 cases was regarded as the standard reference due to reaching a plateau of the Dice coefficients index (Fig. 3d). When we increased training cases from 10 to 50, 50 to 100, 100 to 300, 300 to 400, and 400 to 500, the increase of Dice was 15%, 6.88%, 3.5%, 0.7%, and 0.3%, respectively. Thus, when the sample size is larger than 500, the increase of the Dice coefficient index would be not significant.

Fig. 2.

Optimization of AI system. Dice similarity coefficient of AI system was improved from 62.8% to 89.2% and manual modification time decreased from 68 min to 0 min.

Fig. 3.

AI-based 3D reconstruction modes during optimization phase (a) Vascular trunk and arteriovenous boundaries are misidentified; (b) Vascular trunks and some arteriovenous boundaries are identified; (c) Vascular trunks and arteriovenous boundaries are identified, small vessels were misidentified. (d) Standard model reference, segmental pulmonary arteries and veins are well identified.

The training dataset (500 cases) and validation dataset (40 cases) of the segmentation model in this study were all from patients who used the “Lung 3D reconstruction service” of Linkdoc Information Technology (Beijing) Co., Ltd. All patients read the User Service Agreement and Privacy Policy and signed the informed consent. The data was voluntarily uploaded by the patients, and Linkdoc was explicitly authorized to use the data for research and optimization of the service.

Study design and participants

A prospective comparison study was performed based on a pilot retrospective study at Peking University Cancer Hospital to verify the potential utilization of the AI system. In this retrospective study, we consecutively recruited 113 patients with lung cancer who underwent contrast-enhanced CT scans before surgery and lobectomy or segmentectomy from August 2018 to November 2021. Qualified video data of these surgeries are available for review. The exclusion criteria were pulmonary wedge resection, lack of complete surgical videos, and absence of contrast-enhanced CT scans. Accuracy was verified by reviewing the anatomical structures in the surgical videos. Based on this, we prospectively recruited 139 consecutive patients following the sample size calculation formula. The patients were included if they have completed contrast-enhanced CT scans and they were scheduled to undergo lobectomy or segmentectomy from November 2021 to June 2022. Patients included in the retrospective and prospective cohorts underwent video-assisted thoracic surgery at Peking University Cancer Hospital, and surgical videos were recorded using IMAGE1 S™ 4U (KARL STORZ-ENDOSKOPE (Shanghai) China).

Ethics

The prospective study was registered in the Chinese Clinical Trial Registry (ChiCTR2100050985) and details of the study protocol are available in the Supplementary Material. Informed consent was obtained from all participants in prospective cohort. The whole study was approved by the Peking University Cancer Hospital Institution Ethics Board (2021KT101) and followed the Consolidation Standards of Reporting Trials-Artificial Intelligence (CONSORT-AI).¹⁴ Due to the retrospective nature of the data acquisition and the use of deidentified surgical videos, the informed consent was exempted in retrospective cohort.

Procedures

Accuracy verification in retrospective and prospective studies

In a retrospective cohort study, investigators reviewed anatomical structures from surgical videos and identified detectable lobar/segmental bronchi, segmental arteries, and pulmonary/segmental veins. The accuracy of the broncho-vascular structures was calculated by comparing the anatomical structures from surgical videos with 3D reconstruction models developed by the AI system.

In the prospective cohort, the accuracy was verified by comparing AI-based reconstruction models with intraoperative anatomical structures. Segmental bronchi, arteries, and veins were individually identified for segmentectomies. During lobectomy, investigators dissected the segmental structures to verify the target structures. It is a binary way to judge the anatomical structures in 3D models. To ensure the quality of the reconstructed structure identification, senior surgeons (Wu N, Yan S and Zhang SY) identified segmental bronchi, arteries, and veins during surgery, and checked with the model, another surgeon (Li X) recorded the existence of each structure or not. In segmentectomy, segmental anatomical structures (consisting of segmental bronchi, segmental pulmonary arteries, and veins) were identified directly by surgeons. The same procedure was done in lobectomy for lobar structure. After surgery, senior surgeons would dissect the parenchyma of excised lobes along segmental bronchi and pulmonary vessels to expose segmental or sub-segmental structures for verification. A comparison was made between findings from the surgical anatomy and related structures simulated by the model and recorded in the checklist (Supplementary Material). Anatomical images from the surgical videos or hilum anatomy were retained for confirmation. Meanwhile, we compared the operation time and intraoperative blood loss between the retrospective cohort (without AI assistance) and prospective cohort (with AI assistance) for safety validation.

Comparison between AI system and manual semi-automatic reconstruction system

A classical manual reconstruction software, Mimics®, was chosen for comparison with AI system in terms of efficiency, quality score, and time consumption for 3D reconstruction procedures. The Mimics® system requires users to indicate labels for each of the structures of interest and to indicate thresholds for the grey-value range to be considered. License of Mimics® MIS 24.0 (Leuven, Belgium, Materialize) was obtained from November 4, 2021.

In the prospective cohort, preoperative 3D reconstruction models of the AI system and Mimics® were created using the same CT scans. The AI system finished the work automatically, and the Mimics-based 3D reconstruction was manually completed by a professional trained surgeon (XL). The accuracy of the broncho-vascular structures was calculated by comparing the anatomical structures, and a comparison of accuracy was made between the AI system and Mimics® using anatomical structures as a standard. The time required for the two 3D reconstruction procedures was recorded. In Mimics®, timing commenced from the airway reconstruction and ceased when 3D reconstruction was completed; pairwise comparison was performed to investigate the differences between the two systems. Quality scores based on the visual perception from the AI-based or from Mimics-based models were evaluated by two experienced surgeons according to the quality scoring criteria, and a third researcher adjudicated discrepancies (N.W., S.Z., and S.Y.) (Fig. 4). In quality scoring criteria (Table 1), a score of 1 represents “complete failure of the reconstruction”, 2 represents “misidentification in vascular trunk”, 3 represents “misidentification in vascular branches”, 4 represents “little misidentification in branches”, and 5 represents “no misidentification”. A comparison of scores was made between 3D reconstruction models from the AI system and from Mimics® to evaluate the difference in visual perception.

Fig. 4.

Flow chart of the retrospective and prospective validation study.

Table 1.

Quality scoring criteria.

Scoring	Criteria	Atlas
1	3D reconstruction model cannot be finished.
2	Bronchi or vessels trunks were reconstructed wrongly, causing confusion in clinical practice.
3	Bronchi or vessels trunks were reconstructed correctly, branches were wrongly reconstructed.
4	Bronchi or vessels trunks were reconstructed correctly, few branches were wrongly reconstructed.
5	3D reconstruction model is finished with clear boundary between arteries and veins.

Statistics

The aim of this study was to verify the accuracy and efficiency of the AI system. We pre-analysed 113 patients to estimate the sample size. The accuracy of the AI system was set at 90% (π_T), and the anatomical structures were regarded as the gold reference at 100% (π_C). A difference of 2% (Δ) between the AI-based 3D models and the anatomical structures was tolerable. Z_{(1 − α)} and Z_{(1 − β)} refer to the quantiles of the standard normal distribution (type I error: α = 0.025; type II error: β < 0.2). For the study to have 80% power with a significance level of 5%, the minimum number of patients recruited into the AI system group was 110. To account for a 20% dropout rate, we planned to recruit more than 132 patients into the study following the sample size calculation formula:

$$n_T=n_C=\frac{\left(Z_{1-\alpha }+Z_{1-\beta }\right)\left[{\pi }_C\left(1-{\pi }_C\right)+{\pi }_T\left(1-{\pi }_T\right)/r\right]}{{\left[\left({\pi }_T-{\pi }_C\right)-\Delta \right]}^2}$$

$$n_T=r{n}_C$$

Pulmonary 3D models were completed by AI system and Mimics independently, and accuracy was verified using anatomical structures as a gold standard. The accuracy of 3D models was calculated as the following formula:

$$Accurac{y}_t=\frac{T{P}_t+T{N}_t}{T{P}_t+F{P}_t+T{N}_t+F{N}_t}=\frac{T{P}_t}{T{P}_t+F{P}_t+F{N}_t}=\frac{\sum _{i=1}^{N}T{P}_t\left(i\right)}{\sum _{i=1}^{N}T{P}_t\left(i\right)+F{P}_t\left(i\right)+F{N}_t\left(i\right)}$$

in which, t refers to bronchi, pulmonary arteries, and pulmonary veins; N refers to the number of patients; TP refers to the number of segmental structures (bronchi, pulmonary arteries, and pulmonary veins) that were correctly reconstructed in 3D models. TN refers to the absent segmental structures in anatomical structures, thus TN = 0; FP refers to the number of structures in models while absent in anatomical structures. FN refers to the number of anatomical structures that models failed to display.

Continuous variables with non-normal distributions are presented as medians (interquartile ranges), and continuous variables with normal distributions are presented as means and standard deviations. Pearson's chi-square test was used to assess categorical variables. An analysis without parametric assumptions was carried out for non-normally distributed data sets (Mann–Whitney U test), paired t-tests were applied to compare paired data obeying normal distribution and homogeneity of variance between two groups, and unpaired t-tests were used for comparisons of unpaired data between two groups. The accuracy (proportion) of AI-based and Mimics-based 3D models were compared by using Fisher's exact test. We used R (version 3.6.1; http://www.R-project.org, The R Foundation for Statistical Computing) via RStudio software version 1.2.5033, Python (Python 3.7, Python Software Foundation, Wilmington, DE), and GraphPad Prism software (version 9.0; GraphPad Software Inc.) to analyse and draw diagrams.

Role of the funders

The funder of the study had no role in the study design, data collection, data analysis, data interpretation, or writing of the manuscript.

Results

Characteristics of patients

This retrospective study included 113 patients. The main baseline characteristics of the patients are summarized in Table 2. The median age was 62 years (IQR 56–67 years), and the female-to-male ratio was 1.4:1. Seventy (61.9%) patients underwent lobectomy and 43 (38.1%) underwent segmentectomy. Furthermore, the median length of the nodule was 19 mm (IQR 15–28 mm), and nodules were located in all lobes, with 23 (20.4%) in the right upper lobe, 12 (10.7%) in the right middle lobe, 23 (24.4%) in the right lower lobe, 40 (35.4%) in the left upper lobe, and 15 (13.3%) in the left lower lobe. The median operation time was 145 min (IQR 127–165 min), and the median intraoperative blood loss was 30 mL (IQR 30–50 mL).

Table 2.

Characteristics of the included patients.

Characteristic	Retrospective study	Prospective study	p-value
	n (%)	n (%)
Age, median (IQR)	63 (56–67)	61 (54–66)	0.253
BMI, median (IQR)	24.2 (21.8–26.8)	24.1 (22.4–26.9)	0.655
Gender			0.302
Male	47 (41.6)	48 (34.5)
Female	66 (58.4)	91 (65.5)
Tumor location			0.107
Right upper lobe	23 (20.4)	43 (31.0)
Right meddle lobe	12 (10.7)	11 (7.9)
Right lower lobe	23 (24.4)	27 (19.4)
Left upper lobe	40 (35.4)	38 (27.3)
Left lower lobe	15 (13.3)	20 (14.4)
Surgery type			0.087
No surgery	0 (0)	0 (0)
Wedge	0 (0)	19 (13.7)
Segmentectomy	43 (38.1)	33 (23.7)
Lobectomy	70 (61.9)	87 (62.6)
Tumor size, median (IQR), mm	19.0 (15.0–28.0)	18.0 (13.0–23.0)	0.110
Blood loss, median (IQR), mL	30.0 (30.0–50.0)	50.0 (30.0–50.0)	<0.001
Operation time, median (IQR), min	145.0 (127.0–165.0)	120.0 (102.0–150.0)	0.132

In this prospective study, 139 patients (November 4, 2021–June 1, 2022) were included according to the inclusion criteria. The median age of the 139 patients was 61 years (IQR 54–66 years), with a female-to-male ratio of 1.9:1. Eighty-seven patients (62.5%) underwent lobectomy, 33 (23.7%) underwent segmentectomy, and 9 (6.5%) required conversion to pulmonary wedge resection. Surgical videos of seven (5%) lobectomies and three (2.3%) segmentectomies were missing from the images. The median diameter of the nodule was 18 mm (IQR 13–23 mm), with 43 (31.0%) in the right upper lobe, 11 (7.9%) in the right middle lobe, 27 (19.4%) in the right lower lobe, 38 (27.3%) in the left upper lobe, and 20 (14.4%) in the left lower lobe. The median operation time was 120 min (IQR 102–150 min), and the median intraoperative blood loss was 50 mL (IQR 30–50 mL).

Accuracy verification of AI system

Accuracy verification in retrospective cohort

The anatomical structures from 113 surgical videos were retrospectively reviewed to verify the accuracy of the AI system (Table 3). In total, 82 segmental and 46 lobar bronchi were identified from 113 surgical videos, and the accuracy of the AI system was 100%. Of the 283 segmental pulmonary arteries, 275 segmented arteries were predicted using the AI system with an accuracy of 97.2%, and five missing arteries occurred in the left upper lobes. The accuracy rate of segmental pulmonary veins was 93.2%, and 175 of the 190 detectable segmental veins were predicted (Fig. 5a). Sixteen lobar veins were identified, and the accuracy of the lobar pulmonary veins was 100%.

Table 3.

Accuracy of AI system (AI system vs. surgical anatomy) in retrospective group.

Lobes	Bronchi (n)			Pulmonary arteries (n)			Pulmonary veins (n)
		AI	SA		AI	SA		AI	SA
Right upper (23)	B1	6	6	A1	18	19	V1	20	20
	B2	1	1	A2	17	17	V2	16	16
	B3	3	3	A3	21	21	V3	18	19
Segment		10	10		56	57		54	55
Lobe	B	12	12	A	0	0	V	1	1
Right middle (12)	B4	6	6	A4	12	13	V4	9	10
	B5	6	6	A5	12	13	V5	9	10
Segment		12	12		24	26		18	20
Lobe	B	4	4	A	0	0	V	1	1
Right lower (23)	B6	9	9	A6	18	18	V6	6	7
	B7	3	3	A7	12	12	V7	6	7
	B8	4	4	A8	11	11	V8	5	6
	B9	2	2	A9	9	9	V9	2	3
	B10	1	1	A10	10	10	V10	2	3
Segment		19	19		60	60		21	26
Lobe	B	9	9	A	0	0	V	9	9
Left upper (39)	B1 + 2	12	12	A1 + 2	37	38	V1 + 2	23	24
	B3	8	8	A3	29	31	V3	23	25
	B4	5	5	A4	17	18	V4	17	18
	B5	2	2	A5	12	13	V5	13	14
Segment		27	27		95	100		76	81
Lobe	B	15	15	A	0	0	V	4	4
Left lower (16)	B6	4	4	A6	14	14	V6	3	3
	B7 + 8	5	5	A7 + 8	11	11	V7 + 8	3	3
	B9	4	4	A9	8	8	V9	1	1
	B10	1	1	A10	7	7	V10	1	1
Segment		14	14		40	40		8	8
Lobe	B	6	6	A	0	0	V	1	1
Accuracy∗	100%	82	82	97.2%	275	283	93.2%	177	190
Accuracy#	100%	46	46		0	0	100%	16	16

Note: Accuracy∗ refers to the accuracy of segmental branches of the bronchi, arteries, and veins, and Accuracy^# refers to the accuracy of the lobar bronchi, arteries, and veins. SA refers to surgical anatomy. Bold values refer to the summary of each lobe and all lobes.

Fig. 5.

Accuracy verification and accuracy ranking. (a) Accuracy of AI-based 3D reconstruction with the standard of anatomical structures in retrospective study. (b) Ranking for accuracy of AI-based 3D reconstruction in different pulmonary lobes in retrospective study. (c) Accuracy of AI-based 3D reconstruction with the standard of anatomical structures in prospective study. (d) Ranking for accuracy of AI-based 3D reconstruction in different pulmonary lobes in prospective study.

The accuracy of the AI system in surgical segmental branch identification was ranked by different lobes; the right middle lobe showed the lowest accuracy in segmental artery verification (92.3%), and the right lower lobe showed the lowest accuracy in segmental vein verification (Fig. 5b).

Accuracy verification in prospective cohort

The intraoperative anatomical structures of the 120 patients were used to verify the accuracy of the AI system in clinical practice (Table 4). A total of 336 segmental bronchi from 120 patients were detected, and the AI system predicted 326 segmental bronchi with an accuracy of up to 97.0%. Ten segmental bronchi were missed in the AI-based 3D reconstruction models. Of the 342 segmental pulmonary arteries, 339 were predicted in the AI system with an accuracy of up to 99.1%, and three missing arteries occurred in RA1 and LA6. Furthermore, the accuracy of segmental pulmonary veins was 98.8%, 336 segmental veins were correctly predicted from 340 anatomic segmental veins, and four segmental vein misidentifications occurred in the right lobes (Fig. 5c). Misidentification of the 3D reconstruction models occurred in nine cases, accounting for 7.5% of all cases. No significant difference was observed between cases with and without misidentification in operation time (median, 120 min [IQR, 95.0–156.5] vs. 120 min [IQR, 102.8–150.0]; p = 0.824) and intraoperative blood loss (median, 50 mL [IQR, 30.0–75.0] vs. 50 mL [IQR, 30.0–50.0]; p = 0.617).

Table 4.

Accuracy of AI system (AI system vs. surgical anatomy) in prospective group.

Lobes		Bronchi (n)				Arteries (n)				Veins (n)
		AI	Mimics®	SA		AI	Mimics®	SA		AI	Mimics®	SA
Right upper (37)	B1	30	28	31	A1	32	25	34	V1	30	26	30
	B2	25	27	27	A2	26	25	26	V2	26	27	27
	B3	28	28	29	A3	29	28	29	V3	27	27	28
Subtotal	Right upper	83	83	87	Right upper	87	78	89	Right upper	83	80	85
Right middle (10)	B4	10	10	10	A4	10	10	10	V4	10	10	10
	B5	10	10	10	A5	10	10	10	V5	10	10	10
subtotal	Right middle	20	20	20	Right middle	20	20	20	Right middle	20	20	20
Right lower (26)	B6	22	23	24	A6	24	23	24	V6	24	22	24
	B7	19	19	20	A7	21	18	21	V7	18	16	19
	B8	20	20	21	A8	23	23	23	V8	19	20	20
	B9	20	20	20	A9	20	20	20	V9	20	17	20
	B10	16	16	16	A10	19	19	19	V10	23	22	23
Subtotal	Right lower	97	98	101	Right lower	107	103	107	Right lower	104	97	106
Left upper (32)	B1 + 2	28	26	28	A1 + 2	29	28	29	V1 + 2	25	24	25
	B3	24	24	25	A3	23	20	23	V3	24	22	24
	B4	18	17	18	A4	17	15	17	V4	17	17	17
	B5	15	15	15	A5	14	12	14	V5	16	15	16
Subtotal	Left upper	85	82	86	Left upper	83	75	83	Left upper	82	78	82
Left lower (14)	B6	11	11	12	A6	9	8	10	V6	11	10	11
	B7 + 8	11	10	11	A7 + 8	12	10	12	V7 + 8	12	10	12
	B9	10	10	11	A9	11	8	11	V9	12	8	12
	B10	9	9	9	A10	10	9	10	V10	12	10	12
Subtotal	Left lower	41	40	42	Left lower	42	35	43	Left lower	47	38	47
Total		326	323	336		339	311	342		336	313	340
Accuracy	97.1%		96.1%	100%	99.1%		91.0%	100%	98.8%		92.1%	100%

Note: Accuracy refers to the accuracy of segmental branches of the bronchi, arteries, and veins. SA refers to surgical anatomy. Bold values refer to the summary of each lobe and all lobes.

The accuracy of the AI system was also ranked by lobe, and the right upper lobe showed the lowest accuracy in segmental bronchi (95.4%), segmental arteries (97.8%), and segmental veins (97.6%). The right middle lobe showed the highest accuracy in the segmental bronchi (100.0%), segmental arteries (100.0%), and segmental veins (100.0%) (Fig. 5d).

Safety verification of AI system

Operation time and intraoperative blood loss

Operation time refers to the time of surgery. Surgery in the retrospective cohort was not assisted by 3D reconstruction, which was regarded as the “without AI system supporting” group, and the prospective cohort was regarded as the “with AI system assistant” group. In the retrospective cohort, the median operation time of 113 patients in the retrospective cohort was 145 min (IQR 127–165 min), and the median intraoperative blood loss was 30 mL (IQR 30–50 mL). In the prospective cohort, the median operation time of the 120 patients was 124 min (IQR 106–150 min), and the median intraoperative blood loss was 50 mL (IQR 30–50 mL). Considering the different demands for 3D reconstruction in lobectomy and segmentectomy, we analysed the operative time and intraoperative blood loss using different surgical procedures. For lobectomy, the operation time of 87 patients with preoperative AI system assistance (median 125.0 min, IQR 107.0–150.0 min) was decreased compared with that of 70 patients without AI assistance (median 149.5 min, IQR 127.0–181.8 min) (p < 0.001) (Fig. 6a), while the intraoperative blood loss of lobectomy showed no significant difference (p = 0.201) (Fig. 6b). For segmentectomy, the operation time of 33 patients with the preoperative AI system assistance (median 120.0 min, IQR 101.5–147.5 min) was decreased compared with that of 43 patients without the assistance (median 140.0 min, IQR 126.3–161.3 min) (p = 0.007) (Fig. 6c), while the intraoperative blood loss showed no significant difference (p = 0.834) (Fig. 6d).

Fig. 6.

Comparison of operation time and intraoperative blood loss. (a) Operation time of lobectomy. (b) Intraoperative blood loss in lobectomy. (c) Operation time of segmentectomy. (d) Intraoperative blood loss in segmentectomy.

Efficiency verification of AI system

Accuracy comparison between AI system and Mimics®

In the prospective cohort, the AI-based 3D reconstruction model was compared with the Mimics® model before surgery using CT scans as a standard, which indicated a higher accuracy in pulmonary blood vessel simulation from the AI model over the Mimics® model (Supplementary Material). Furthermore, the accuracy of AI-based and Mimics-based 3D reconstruction models was compared using anatomical structures as a standard. The AI system showed an accuracy similar to that of Mimics® in segmental bronchi (97.1% vs. 96.1%, p = 0.336) and outperformed Mimics® in segmental artery prediction (99.1% vs. 91.0%, p < 0.0001) and segmental vein prediction (98.8% vs. 92.1%, p < 0.001) (Fig. 7a).

Fig. 7.

Efficiency comparison of AI system and Mimics. (a) Accuracy verification of AI system and Mimics by anatomical structures standard. (b) Quality scores of AI-based and mimics-based 3D reconstruction vision demos. (c) One patient received left upper lobectomy. Of which segmental bronchi, segmental arteries, and segmental veins were labelled, the quality scores of AI system and Mimics were 5 and 4, respectively. The Mimics-based reconstruction model lose some small branches of bronchi and vessels. (d) Time consumption of AI system and Mimics.

Quality scoring of AI system and Mimics®

The surgeons graded the quality score based on the visual perception of the reconstructed image according to the quality scoring criteria (Table 1). Owing to the optimization of the AI system and Mimics®, none of the cases were scored as 1 or 2. Ninety-eight AI-based 3D models scored 5 (70.5%), compared with 63 Mimics-based 3D models scored 5 (45.3%). In the field of reconstruction models scored as 3 or 4, Mimics-based 3D models occupied more cases than AI-based 3D models (3:12.2% vs. 1.4%; 4:42.5% vs. 28.1%). A significant difference was observed in the quality scores between the two types of 3D reconstruction vision demos (p < 0.001) (Fig. 7b). The AI-based model and Mimics-based model of one patient who underwent left upper lobectomy are displayed here (Fig. 7c).

Time consumption of AI system and Mimics®

The median total reconstruction time of AI system was 6.8 min (IQR 5.5–8.1 min), less than 21 min (IQR 17–28 min) of the manual reconstruction time in Mimics® with a significant difference (p < 0.001) (Fig. 7d).

Discussion

With the development of computer technology, interactive 3D reconstruction has proven useful in enhancing surgeons' experience with specific anatomic structures, especially in patients with anatomic variations.^1,2 In recent hepatic surgery, 3D reconstruction has gradually been used to visualize and intuitively display variations in intrahepatic blood vessels. It can also provide a convenient and accurate method for liver volume calculation, virtual simulation surgery, and surgical navigation.³ Several manual software programs have been developed for the application of 3D reconstruction systems in thoracic surgery. In the context of an increasing number of thoracic surgeries for lung resection, manipulation skills and the time consumption of manual software would curb surgeons’ enthusiasm for completing 3D reconstructions. As AI technology has evolved in disease diagnosis and treatment since it was proposed in the 1960s,¹⁵ it has been applied in various medical fields to replace part of the manual work. Therefore, we propose the fully automated AI-based 3D reconstruction system and validate its accuracy and efficiency by comparison with a classical manual system for future clinical use.

Semi-automatic reconstruction tools, represented by Mimics®, are based on threshold-based segmentation methods¹⁶ from stacks of contrast-enhanced 2D CT scans, achieving an actionable but laborious procedure in completing the 3D reconstruction models. In this study, we constructed the AI system based on deep learning, which is a fully automated algorithm that can efficiently complete 3D reconstruction with non-contrast or contrast-enhanced CT scans. This AI system has high modularity, in which the modules are designed using specific segmentation algorithms for different pulmonary tissues, and each module can run and be optimized independently. Relying on data-driven optimization for deep learning¹⁷ and the feedback error learning concept in clinical practice, this AI system could run smoothly with little human intervention. It also had excellent generalization ability by using a fully automated algorithm in both contrast-enhanced and non-contrast-enhanced CT. Although high modularity requires considerable data collection and annotation by experienced doctors for multi-task learning during the training process, it can greatly enhance the efficiency after optimization. The AI system can also automatically smoothen the 3D reconstruction model in the post-processing module. However, ensuring safety in clinical practice is vital for the AI system, and accuracy verification is still lacking, especially when compared with human anatomical structures.

The Dice similarity coefficient was used to evaluate the segmentation performance of the medical images by calculating the similarity of the models and CT images with a value between 0 and 1. In a previous study, the accuracy of the 3D reconstruction model was verified by comparison with CT, in which the fully automated algorithm reached a stable Dice similarity of 85%.¹⁸ To accurately support clinical practice, a prospective comparison study between AI-based 3D models and anatomical structures is urgently needed. As the AI system reached a reasonably stable Dice similarity coefficient of 89.2%, we verified its accuracy of AI system by comparing the AI-based 3D reconstruction model with intraoperative anatomical structures. In retrospective studies, limited data on segmental branches from surgical videos can be used for accuracy verification, especially in lobectomy. Segmental pulmonary veins were relatively poorly predicted in a retrospective study, with an accuracy of 93.2%, and misidentification always occurred in the left upper lobe owing to its complex anatomical structures with small branch and blood vessel variations.¹⁹ Thus, this prospective study was designed with intraoperative accuracy verification to reach a highly reliable conclusion. Ranking accuracy showed that misidentification always occurred in the right upper lobes because unrecognized arteriovenous crossover and different anatomic subtypes existed in the right upper lobes for different positions of the central veins,²⁰ which should be noted to avoid major bleeding due to blood vessel injury. In this study, we also found that segmental structure misidentification in a 3D model for preoperative planning had little impact on operation time and intraoperative blood loss, suggesting a potential safety profile for clinical application. Considering the limited sample size in this regard, a large sample size study is needed for future safety validation.

The AI system may significantly reduce the reconstruction time compared with the classical manual reconstruction system. The time consumption of the AI system consisted of the automated reconstruction time and clinical manipulation time. In clinical practice, automatic reconstruction is initiated when patients finish their CT examination, and no extra manipulation is needed as surgeons review the AI-based 3D reconstruction models. Thus, the AI system has an overwhelming advantage in saving surgeons' manipulation time because even skilled surgeons need to annotate the anatomical structures in CT layer-by-layer in the classical manual system. Quality score evaluation by surgeons according to the criteria suggested that the AI system can also provide 3D reconstruction vision demonstration with better visual perception, which is mostly related to the identification of blood vessel boundaries. The merit of the AI system is that it provides a landscape reconstruction view of pulmonary structures, which may localize to any interesting area and scrutinize detailed structures for surgical planning. However, for manual systems, regional structure reconstruction is a common procedure in most practice because whole-scale reconstruction with manual labelling is very time-consuming.

To our knowledge, there are very few prospective studies that have been designed to verify the accuracy of an AI system. Here, we compared AI-based 3D phantom structures with intraoperative surgical anatomical structures in both retrospective and prospective cohorts. Combined with the surgeon's experience, this AI system can effectively avoid misidentification in intraoperative decision-making. Thus, the operation time in both lobectomy and segmentectomy can be reduced because the preoperative 3D model would help surgeons individually recognize vascular variations and their position, orientation, and relationship with other blood vessels for safer surgery.⁶ Previous studies have shown that careful preoperative planning in thoracic surgery is essential to reduce intraoperative complications.21, 22, 23 Especially in anatomical lung resection, it is vital to avoid injury to hilar structures with complex vascular structures.⁶

AI-based 3D models can also simplify the procedures of training less experienced thoracic surgeons. The traditional training process is a combination of the self-learning process and the mentoring process by seniors. In an era of computer science, steric knowledge of anatomic structures can be mastered with the aid of semi-automatic and fully-automatic systems. The junior surgeon may use the AI system to establish a landscape picture of any lobar or segmental anatomy quickly, like the guide from a mentor, and then use the semi-automatic system to label vascular and bronchi from CT images and recognize or identify the individual vascular-bronchi structures or variants. During this process, a quick perception of the targeted pulmonary structure would be enhanced and then this perception is validated or guided by senior surgeons during the operation. Recent articles showed that AI technology was superior in skill transfer compared with remote expert instruction.²⁴ In our department, young fellows gain knowledge of anatomic structures quickly by annotating these complicated anatomical structures during their initial stage of practice. When they fluently master the technique, they will transfer to the fully-automatic AI system to save time on reconstruction. We believe this combination will help their training process.

The present study has several limitations. Although the AI system has reached a stable level with considerable data collection, CT artifacts still hinder its identification ability and reconstruction performance. Then, 3D-VNet was used to enhance the identification ability, and the attention mechanism was adopted to improve its ability to extract global semantic information to reduce disturbance information. Additionally, there was a paucity of previous direct comparative studies with anatomical structures, and the sample size (n = 139) was calculated according to the results of a retrospective study (n = 113); thus, the confounding effects of anatomical structure variation of limited sample size cannot be ruled out. Finally, this was a single-centre study consisting of retrospective and prospective studies, and the results were subjective to the inherent shortcomings of the single-centre study. Patients in preparation for lung cancer surgery were enrolled, which may introduce a selection bias. Thus, data from larger patient populations from multiple centres are required to confirm the accuracy, efficiency, and safety of the AI system.

In conclusion, this comprehensive study of the AI system provided evidence of its high accuracy in predicting pulmonary anatomic structures with high efficiency. With the assistance of the AI system, the operation time was significantly reduced, with no significant influence on intraoperative blood loss.

Contributors

N.W. and S.Y. supervised the study. N.W., S.Y., X.Li, S.Z., and L.T. conceived of and designed the study. X.Li, G.G., and X.L. trained and developed the artificial intelligence model. X.Li did the statistical analysis. X.Li and S.Z. wrote the drafted report. N.W., S.Y., S.L., Y.W., X.C., and S.W. critically revised the manuscript. B.L., Y.T., and B.X. organized and screened patients. All authors had access to all the raw datasets. X.Li, S.Z., and D.Z. verified all the data. All authors revised the report and approved the final version before submission.

Data sharing statement

All de-identified participants’ data are available with the article and its Supplementary material, or are available from the corresponding authors (Email: nanwu@bjmu.edu.cn) upon reasonable request.

Declaration of interests

The authors declare no competing interests.