Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2022 Apr 1.
Published in final edited form as: J Vasc Interv Radiol. 2021 Dec 16;33(3):324–332.e2. doi: 10.1016/j.jvir.2021.12.017

Quantitative Automated Segmentation of Lipiodol Deposits on Cone-Beam CT Imaging Acquired during Transarterial Chemoembolization for Liver Tumors: A Deep Learning Approach

Rohil Malpani 1, Christopher W Petty 1, Junlin Yang 1, Neha Bhatt 1, Tal Zeevi 1, Vijay Chockalingam 1, Rajiv Raju 1, Alexandra Petukhova-Greenstein 1, Jessica Gois Santana 1, Todd R Schlachter 1, David C Madoff 1, Julius Chapiro 1, James Duncan 1, MingDe Lin 1
PMCID: PMC8972393  NIHMSID: NIHMS1791267  PMID: 34923098

Abstract

Purpose:

To show that a deep learning (DL)–based, automated model for Lipiodol (Guerbet Pharmaceuticals, Paris, France) segmentation on cone-beam computed tomography (CT) after conventional transarterial chemoembolization performs closer to the “ground truth segmentation” than a conventional thresholding-based model.

Materials and Methods:

This post hoc analysis included 36 patients with a diagnosis of hepatocellular carcinoma or other solid liver tumors who underwent conventional transarterial chemoembolization with an intraprocedural cone-beam CT. Semiautomatic segmentation of Lipiodol was obtained. Subsequently, a convolutional U-net model was used to output a binary mask that predicted Lipiodol deposition. A threshold value of signal intensity on cone-beam CT was used to obtain a Lipiodol mask for comparison. The dice similarity coefficient (DSC), mean squared error (MSE), center of mass (CM), and fractional volume ratios for both masks were obtained by comparing them to the ground truth (radiologist-segmented Lipiodol deposits) to obtain accuracy metrics for the 2 masks. These results were used to compare the model versus the threshold technique.

Results:

For all metrics, the U-net outperformed the threshold technique: DSC (0.65 ± 0.17 vs 0.45 ± 0.22, P < .001) and MSE (125.53 ± 107.36 vs 185.98 ± 93.82, P = .005). The difference between the CM predicted and the actual CM was 15.31 mm ± 14.63 versus 31.34 mm ± 30.24 (P < .001), with lesser distance indicating higher accuracy. The fraction of volume present ([predicted Lipiodol volume]/[ground truth Lipiodol volume]) was 1.22 ± 0.84 versus 2.58 ± 3.52 (P = .048) for the current model’s prediction and threshold technique, respectively.

Conclusions:

This study showed that a DL framework could detect Lipiodol in cone-beam CT imaging and was capable of outperforming the conventionally used thresholding technique over several metrics. Further optimization will allow for more accurate, quantitative predictions of Lipiodol depositions intraprocedurally.

This study and past studies have made use of cone-beam computed tomography (CT) for intraprocedural imaging during interventional oncologic procedures. However, the inherent nature of cone-beam CT is somewhat limiting, in that it lacks a standardized calibration of image density. This has restricted the ability to automatically characterize Lipiodol (Guerbet Pharmaceuticals, Paris, France) deposition quantitatively. One previous study (1) using cone-beam CT to visualize and characterize Lipiodol found that cone-beam CT is significantly more accurate than fluoroscopy at assessing Lipiodol retention patterns. Although cone-beam CT has potential as a tool for the imaging of Lipiodol, there is room for optimization. Deep learning (DL) is a highly promising avenue for optimizing the power of imaging modalities such as cone-beam CT. A plethora of recent research has used DL to characterize and quantify chemoembolization and subsequently predict long-term patient outcomes (2). This study sought to further this research.

Segmenting Lipiodol on cone-beam CT scans can be categorized under a field of problems known as “computer vision” tasks. In the past decade, enormous progress has been made in computer vision, thanks to DL algorithms called convolutional neural networks (CNNs). CNNs are an example of supervised learning; they accept a data set of inputs (cone-beam CT images, in this study) and outputs (here, binary segmentations of Lipiodol) and fit a model mapping one to the other. The feature finding nature of CNNs has the added benefit of being able to identify unnatural artifacts, learning to account for them, or, in other scenarios, recreating whole images without them (3).

The purpose of this study was to develop a DL-based solution that overcomes the inherent limitations of cone-beam CT and achieve automated and quantitative Lipiodol segmentation. Specifically, the aim was to apply a DL approach to automatically segment Lipiodol on cone-beam CT obtained intraprocedurally during conventional transarterial chemoembolization, compare this segmentation with the existing statistical methods for Lipiodol segmentation that include semiautomatic thresholding techniques, and compare these results with ground truth segmentations produced by a board-certified radiologist. The findings of this study have the potential to improve upon existing methods for the quantification of Lipiodol. This, in turn, would allow physicians to feel comfortable relying heavily on cone-beam CT imaging and using obtained cone-beam CT data to make predictive inferences about treatment success and even patient outcome.

MATERIALS AND METHODS

Patient Selection

This study was conducted in compliance with the Declaration of Helsinki on ethical principles for medical research involving human subjects and approved by the Yale University institutional review board (IRB). This was a post hoc analysis. Imaging and data used in this study were obtained from patients who were enrolled in 2 separate prospective clinical trials with a specified imaging protocol and standardized cone-beam CT acquisition that followed strict norms, the application of an open-trajectory system, and the use of a standardized Lipiodol emulsion. The data from the clinical trials were then retrospectively used in this post hoc analysis. Upon completion of the original trials and after meeting all enrollment end points, the imaging data from those trials were reviewed and used in a retrospective fashion. An IRB exemption for data review was obtained. This study involved analysis of the cone-beam CT images acquired intraprocedurally during a conventional transarterial chemoembolization procedure. Patients in this study included those with a diagnosis of hepatocellular carcinoma (HCC) or other solid liver tumors (non-HCC, intrahepatic cholangiocarcinoma, or liver-predominant metastatic disease). The patients were aged ≥18 years and with Child-Pugh class A or B liver function (only Child-Pugh class A for patients without HCC) and had an Eastern Cooperative Oncology Group performance status of 0–2. The exclusion criteria for this study were as follows: (a) advanced cardiac or severe systemic disease (defined as clinically relevant extrahepatic tumor burden outside the liver and nonliver dominant disease); (b) known allergy to Lipiodol, poppy seed oil, or iodinated contrast agents; (c) breastfeeding/pregnant patients; (d) main portal vein thrombosis; (e) patients with contraindications to chemotherapy agents used in the conventional transarterial chemoembolization procedure (doxorubicin and Mitomycin-C [Pharmacia & Upjohn, Peapack, New Jersey]); and (f) a lack of good-quality cone-beam CT imaging free of severe motion artifact.

Patient Demographics

This study included 36 patients. All patients had undergone a conventional transarterial chemoembolization procedure that included intraprocedural cone-beam CT acquisition. All procedures were performed between 2012 and 2018. Six patients were excluded from this study due to poor cone-beam CT image quality such as severe motion artifacts (ie, streak artifacts). Seven women and 29 men were included in this study; the average age of this cohort was 61 years ± 8.3.

Conventional Transarterial Chemoembolization Procedure and Cone-Beam CT Acquisition

The conventional transarterial chemoembolization procedure was performed according to standard, IRB-approved protocols by 3 board-certified interventional radiologists with 5–20 years of experience in hepatic interventions (T.R.S., D.C.M., and a non-author interventional radiologist performed the procedures). Lobar or selective conventional transarterial chemoembolization was performed in this study. Details on the conventional transarterial chemoembolization procedure and intraprocedural cone-beam CT acquisition are presented in Appendix A (available online on the article’s Supplemental Material page at www.jvir.org). Terminology used to describe transcatheter therapies were reported using society guidelines (4).

Image Preprocessing and Lipiodol Segmentation

The liver and Lipiodol deposition regions were segmented in a 3-dimensional (3D) form on the raw cone-beam CT using a semiautomated quantification and segmentation software (GeoBlend 3D; Philips Healthcare, Best, the Netherlands) by a board-certified radiologist. A well-established and prevalidated semiautomated technique was used, which has been proven to be reliable by several groups on prior occasions, even if 1 reader is involved (5–8). All segmentations were performed by an interventional radiologist with 18 years of experience who did not perform the transarterial chemoembolization, and the Lipiodol deposition segmentation served as the ground truth. The radiologist was blinded to the results from the DL protocol and the periprocedural imaging. The radiologist was tasked with reading the cone-beam CT image itself and drawing margins on the basis of that image. Figure 1 shows the image preprocessing pipeline.

Figure 1.

Figure 1.

Open in a new tab

The depiction of the image preprocessing workflow: original cone-beam computed tomography (CT) (a) to the liver-segmented cone-beam CT (b) to the radiologist-segmented Lipiodol (c), which is used as the ground truth for assessing the performance of the U-net’s Lipiodol deposition prediction (d). This figure goes over the imaging preprocessing workflow going from the original cone-beam CT (a) to the liver-segmented cone-beam CT (b) to the radiologist-segmented radiology (c) (which serves as the ground truth). This is fed into the U-net to evaluate the Lipiodol predictions of the model (d). CBCT = cone-beam computed tomography.

CNN Architecture

The architecture of CNN used is illustrated in Figure 2. The structure of this architecture is commonly referred to as a U-net. As with all CNNs, the data were contracted via a series of convolutions. U-nets of this kind are commonly used in biomedical imaging because they yield accurate segmentations, given smaller training sets (9–11). Further details on the CNN architecture, loss function, model training, Lipiodol mapping, image augmentation, and K-fold cross validation are presented in Appendix A (available online at www.jvir.org).

Figure 2.

Figure 2.

Open in a new tab

Architecture of the U-net that generates the Lipiodol segmentation mask. This figure presents an overview of the entire deep learning pipeline going from the unsegmented cone-beam computed tomography (CT) to the liver-segmented cone-beam CT, which is subsequently used as the input to the U-net. The output of the U-net is initially a Lipiodol probability map, which is subsequently converted to the model’s prediction of the Lipiodol segmentation map. This is compared with the radiologist-segmented Lipiodol (which serves as the ground truth) via metrics such as dice similarity coefficient, mean squared error, and center of mass. CBCT = cone-beam computed tomography; CM = center of mass; DSC = dice similarity coefficient; MSE = mean squared error.

Thresholding Technique

To assess the U-net model autosegmentation of Lipiodol on cone-beam CTs after conventional transarterial chemoembolization, the accuracy of its predictions (compared with the radiologist-segmented ground truth) was compared with the “thresholding technique” that has been used on CT scans for Lipiodol delineation. The ground truth in this case was the 3D-volumetric manual segmentation of Lipiodol on livers after conventional transarterial chemoembolization performed by a board-certified radiologist. This was performed using a semiautomated quantification and segmentation software (GeoBlend 3D).

In intensity thresholding used in the “thresholding technique,” essentially window leveling in which Lipiodol is considered present on CT imaging on pixels with >270.2 Hounsfield units (HUs) (12) is performed and, as a technique, is commonly used in various fields of biomedical research (13,14). Both the U-net (Fig 2) and thresholding technique results were initially compared with the ground truth to obtain the segmentation evaluation metrics. These were subsequently compared with each other to determine which approach was superior.

In this comparison, Lipiodol was considered present in an area if the intensity of that area was greater than a certain number of standard deviations above the mean intensity. The number of standard deviations was determined by starting at 1 and incrementally raising that number by 0.5 until the best average dice similarity coefficient (DSC) score was achieved. This was found to be 3 standard deviations above the mean. A depiction of the histogram of cone-beam CT pixel intensities used in the thresholding technique cutoff determination can be found in Figure 3.

Figure 3.

Figure 3.

Open in a new tab

An example of the pixel intensity histogram for a liver-segmented cone-beam computed tomography (CT). This figure depicts the pixel intensity histogram for a liver-segmented cone-beam CT from this study with the mode, 1 standard deviation (SD), and 2 SD above the mode overlaid on the histogram. This showcases the method used to determine the cutoff pixel intensity values used for the thresholding technique.

Segmentation Evaluation Metrics and Statistical Analysis

The performances of the U-net model and thresholding method were compared with the ground truth, radiologist Lipiodol segmentations using DSC, mean squared error (MSE), center of mass (CM), and fractional volume ratios (fraction of the total volume present in the prediction or the threshold divided by the volume in the ground truth). These metrics have been used extensively in the literature (15,16). The CM metric considered each pixel in which Lipiodol is present as 1 point of mass. The result for this metric was the distance (in millimeter) between the CM on the predicted/threshold mapping and the ground truth. The intuition for the CM test was that it would reward predictions with a similar size and location as the ground truth while de-emphasizing the necessity of a pixel-by-pixel overlap between the corresponding regions. This is because checking for direct overlap not only is less relevant clinically but also can often be inaccurate in manual/semiautomated segmentations by human operators. Independent 2-sample 2-tailed t tests were used to compare the aforementioned metrics between the model’s prediction and the thresholding technique predictions. A P value of ≤.05 was considered statistically significant.

RESULTS

Image Analysis

The success of the experiment was determined by the improvement of the neural network’s predictions compared with the results of the intensity threshold method. The results were averaged over 6 folds and are shown in the Table. The examples of liver-segmented cone-beam CT along with the ground truth, model’s predictions, and thresholding technique are presented in Figure 4. The first 2 metrics used to test the results were the DSC test and MSE. The results for the model prediction versus the threshold technique were as follows: DSC, 0.65 ± 0.17 versus 0.45 ± 0.22 (P < .001), and MSE, 125.53 ± 107.36 versus 185.98 ± 93.82 (P = .005).

Table.

Quantitative Performance Measurements of the U-Net and Threshold Methods versus the Radiologist Lipiodol Segmentation Ground Truth

Mode evaluation metrics Threshold compared with the ground truth U-net compared with the ground truth Two-tailed t test between threshold and U-net
MSE 185.98 ± 93.82 125.53 ± 107.36 P = .0046
CM* (mm) 31.34 ± 30.24 15.31 ± 14.63 P = .000012
Fractional volume ratios† 2.58 ± 3.52 1.22 ± 0.84 P = .048
DSC 0.45 ± 0.22 0.65 ± 0.17 P = .000025
Open in a new tab

Note–The results are averaged over all patients. Note the improved performance of the U-net over the threshold method in predicting Lipiodol deposition on conventional transarterial chemoembolization. Score evaluation: MSE and CM: lower score indicates higher accuracy. Fractional volume ratios and DSC: closer to 1 indicates higher accuracy.

CM = center of mass; DSC = dice similarity coefficient; MSE = mean squared error.

*

CM values represent the distance from the Lipiodol’s center of mass.

†

Fractional volume represents the fraction of total volume predicted over the ground truth. ie, Fractional volume = (predicted volume of Lipiodol)/(ground truth volume of Lipiodol).

Figure 4.

Figure 4.

Open in a new tab

Four cases showcasing the input cone-beam computed tomography (CT) provided versus the radiologist-segmented ground truth, threshold technique output, and model’s prediction. This figure compares the original cone-beam CT with liver segmentation to binary Lipiodol segmentations made by a radiologist (the ground truth), thresholding technique, and model output. As supported by the data, the radiologist-segmented ground truth more closely resembles the model output. CBCT = cone-beam computed tomography.

The difference between the model’s predicted CM and ground truth’s CM were smaller than the difference between the thresholding technique’s CM and the ground truth’s CM (15.31 mm ± 14.63 vs 31.34 mm ± 30.24, P <.001), with a lower value indicating higher accuracy. The final test performed compared the total volume of Lipiodol predicted by the model or the thresholded values with the total volume of deposited Lipiodol in the ground truth. The fraction of volume present (predicted Lipiodol volume/ground truth Lipiodol volume) was 1.22 ± 0.84 versus 2.58 ± 3.52 (P = .048) for the current model’s prediction and threshold technique, respectively.

DISCUSSION

The main finding of this study was that this DL model can segment Lipiodol deposition automatically on intraprocedural cone-beam CT. The DL method performed better than conventionally used thresholding techniques. This fact is particularly relevant because the nonstandardized/noncalibrated nature of cone-beam CT has prevented its increased utilization in conventional transarterial chemoembolization procedures for procedural efficacy determination and newer DL technologies can help alleviate this problem.

The traditional thresholding techniques used in clinical practice involves the utilization of a preset value of pixel/voxel intensity as the cutoff point to delineate a region of interest (ie, the presence of Lipiodol, in this case). In the literature, thresholding has been shown to be effective in automatically delineating Lipiodol deposition on CT images after conventional transarterial chemoembolization (17). Because CT images are composed of standardized/absolute values of voxel intensities via HUs, this allowed for standardized cutoff values to be used to further separate low-density from high-density Lipiodol (17). However, the corresponding metric to HUs in the case of cone-beam CT imaging was quantitative gray units that were not comparable to the former (18). Cone-beam CT is more sensitive to motion because acquisition times are on the order of 4–10 seconds, whereas those of multidetector CT are subseconds. The nature of cone-beam x-ray generation results in more x-ray scatter than fan-beam multidetector CT, leading to diminished image contrast. Furthermore, cone-beam CT image acquisition typically captures 220°–240° around the patient, creating a partial/limited projection view, whereas multidetector CT captures a full 360° around the patient. All these factors together result in more image noise and cupping/beam hardening in cone-beam CT compared with multidetector CT (19). This leads to more difficulty in the automatic processing of cone-beam CT imaging. Automatic segmentation of Lipiodol poses multiple advantages over manual segmentations (20,21), namely, removal of reader-dependent bias and the difficulties owing to the varying distribution patterns of Lipiodol. Although there are image acquisition and quality downsides associated with cone-beam CT imaging, cone-beam CT has an advantage over multidetector CT, in that the imaging can be acquired intraprocedurally, thus allowing for modification of treatment while the patient is still on the table (22). Due to the nonstandardized nature of cone-beam CT, the use of the threshold technique generates Lipiodol segmentations that are very diffuse and markedly different from the ground truth (Fig 4).

Previous works in the literature have successfully achieved cone-beam CT to multidetector CT or magnetic resonance (MR) imaging registration (5–7,23–27). These studies have shown success in liver, brain, and head and neck imaging and in comparing Lipiodol and tumor across the different modalities. These studies have also used cone-beam CT to CT registration for radiation therapy planning imaging (26,27). Semiautomatic segmentation of Lipiodol on cone-beam CT has been successfully performed in the literature as well (5–7,23). These studies used tumor-to-liver contrast, one of the comparison metrics along with the volume of tumor or Lipiodol (6). However, this study was only in a 2-dimensional form. Studies in 3D were able to successfully show statistically significant overlaps in the tumor and Lipiodol volume on cone-beam CT and multidetector CT (7). However, this study did not evaluate the distribution patterns of the Lipiodol on cone-beam CT versus multidetector CT. One study (23) successfully demonstrated a novel registration method for accurate assessment of microwave ablation outcomes of hepatic malignancies between contrast-enhanced cone-beam CT and multidetector CT imaging. Another study (5) showed strong agreement in tumor volume on contrast-enhanced MR imaging and cone-beam CT imaging in the case of hepatic malignancies. However, this study did not investigate distribution metrics and focused on volume as the outcome metric. The current study’s approach to Lipiodol quantification was different because it used structural DL-based techniques to extract the signal rather than deterministic image intensity thresholds.

DL techniques have been used in the past to successfully generate a synthetic CT image from cone-beam CT imaging (26,27). These techniques remove image artifacts and noise and correct for the intensity difference between CT and cone-beam CT imaging to create synthetic CT scans that are similar to multidetector CT scans obtained for comparison.

U-net is a type of convolutional network often used in biomedical imaging for classification tasks (9,27,28). Compared with other network architectures used for this purpose, a U-net requires fewer annotated images and has a faster run time (9). The use of U-net in the radiology literature has been well documented for studies involving artificial intelligence/machine learning for the reasons outlined earlier (9,29).

In this study, very few parameters of the standard U-net model were changed. It is the clinical use and image postprocessing rather than the model itself that added to existing literature. In the interest of improving model performance, the optimization of various model parameters and image postprocessing factors was attempted as well. Some of these empirical efforts did not result in appreciable improvement in model performance. Examples included testing a variety of loss functions, adding more complex data augmentation techniques beyond image rotation and intensity scaling, testing optimizers other than Adam (name derived from Adaptive Moment Estimation) such as stochastic gradient descent or ascent, and increasing the number of epochs beyond 1,500.

In the initial stages of this study, mapping the cone-beam CT images to the 24-hour post-procedure multidetector CT images obtained after conventional transarterial chemoembolization was attempted to provide secondary data points (via the quantified Lipiodol deposition patterns). However, these approaches were not used for the final study because of real-world challenges in registration between the cone-beam CT and CT imaging. For example, cone-beam CT has a limited field of view, and especially for patients with larger body habitus, there would be image acquisition limitations with truncation of the liver. Furthermore, because DSC scoring is a pixel-by-pixel comparison, the dynamic and unset pattern washout of Lipiodol from the liver over the time span between cone-beam CT and multidetector CT acquisition challenges registration (30). Thus, comparing the Lipiodol deposition in the same tumor at 2 different (nonadjacent) time points on the 2 imaging modalities was reserved for future work.

This study has some limitations. The analyzed cohort of patients enrolled included a heterogeneous cohort of primary and secondary liver tumors, which limited a subgroup analysis. A small number of female study participants limited a sex-based subgroup analysis as well. A slight limitation was the relative difficulty to manually segment Lipiodol on cone-beam CT (especially when the coverage by the injected Lipiodol plus chemotherapeutic agent was diffuse and not concentrated around 1 focus). This led to some minor inconsistencies in the training/testing data being fed into the model. This was partially alleviated by utilizing experienced and board-certified radiologists to conduct the segmentations. No subgroup analysis was undertaken to determine model performance based on tumor characteristics and morphology. Finally, the DL-generated Lipiodol map was not compared with preprocedural MR/CT imaging. This could lead to insights into whether DL methods better predicted tumor response than manual segmentation.

Additionally, more studies should explore the relationship between Lipiodol density and its spatial distribution with model performance. With multiple studies showing the correlation of tumor Lipiodol coverage with tumor response, further studies in the area could provide valuable prognostic information as well (20,30–32). The translation targets of this work into the clinic could be on the C-arm cone-beam CT workstation and the picture archiving and communication system (PACS). In the former, the same graphics processing units used for the cone-beam CT image reconstruction could be used for inference, and the result can be displayed as an overlay on the intraprocedural imaging (cone-beam CT, fluoroscopy, and digital subtraction angiography) for real-time feedback of Lipiodol deposition. Implementation on PACS has the advantage of more processing power for inference (especially if leveraging cloud deployment), easier ability to bring in prior imaging for reference, and easier deployment. One roadmap for translation could be to deploy on the PACS first to refine and retrain (continual learning) the U-net with additional data (imaging and radiologist-revised Lipiodol prediction maps: new ground truth) and, once the U-net is more mature, to subsequently deploy on the C-arm (33).

In summary, this study showed that a DL framework is capable of outperforming the conventionally used thresholding technique. Future work could allow for more accurate predictions intraprocedurally. This would allow for the modification of treatment at the time of treatment while the patient is still on the table.

RESEARCH HIGHLIGHTS.

  • A deep learning framework achieved automated, accurate, and quantitative segmentations of Lipiodol (Guerbet Pharmaceuticals, Paris, France) on cone-beam computed tomography (CT) during conventional transarterial chemoembolization.

  • The model’s prediction more closely resembled the true deposition of Lipiodol in shape, size, and location than the routine clinical manual thresholding techniques.

  • Model evaluation metrics are proposed that are clinically relevant for Lipiodol localization beyond their use in standard image segmentation analysis, including the dice similarity coefficient, mean squared error, center of mass, and fractional volume ratios.

  • This deep learning approach may overcome limitations of the nonstandardized/noncalibrated nature of cone-beam CT imaging.

STUDY DETAILS.

Study type:

Retrospective, observational, descriptive study

Level of evidence:

3 (SIR-C)

ACKNOWLEDGMENTS

The authors thank Eliot Funai, CCRP, for help with study protocols and imaging retrieval.

N.B. and V.C. report grant support from the American Association of Physicists in Medicine, during the conduct of the study. J.C. reports grant support from the Society of Interventional Oncology, Yale Center for Clinical Investigation, the National Institutes of Health (NIH R01CA206180), Boston Scientific, and BTG and personal fees from Philips Healthcare, outside the submitted work. D.C.M., J.C., and T.R.S. report grants/personal fees from Guerbet Pharmaceuticals, during the conduct of the study. M.L. is a current Visage Imaging, Inc., employee and stockholder; is a former Philips Research North America employee, outside the submitted work; reports grant support from the National Institutes of Health (NIH R01CA206180), during the conduct of the study; and is a board member of Tau Beta Pi Engineering Honor Society. None of the other authors have identified a conflict of interest.

ABBREVIATIONS

CM

center of mass

CNN

convolutional neural network

3D

3-dimensional

DL

deep learning

DSC

dice similarity coefficient

HCC

hepatocellular carcinoma

HU

Hounsfield unit

IRB

institutional review board

MSE

mean squared error

PACS

picture archiving and communication system

APPENDIX A

Further Methodological Details of this Study

Conventional Transarterial Chemoembolization Procedure.

First, using multiple angiographic steps, the hepatic vasculature feeding the tumor was identified. Upon catheterization of the selected vessel, chemoembolization was performed. This emulsion was created using 10 mL of Lipiodol (Guerbet Pharmaceuticals, Paris, France) and 10 mL of chemotherapeutic agent (with 50 mg of doxorubicin and 10 mg of Mitomycin-C [Pharmacia & Upjohn, Peapack, New Jersey]), mixed thoroughly using a push-and-pull method to obtain a homogeneous stable solution. The volume of chemoembolic emulsion injected depended on patient factors and tumor size. Following this injection, 1 vial of 100–300-μm gelatin-coated tris-acryl microspheres (Merit Medical, South Jordan, Utah) was administered to achieve the embolization end point. The water-in-emulsion was created via Lipiodol to chemotherapy ration of slightly greater than 1:1. This ratio was adjusted if its viscosity decreased the arterial flow rate. The technical end point of this procedure was considered to be achieved when the entire dose of chemotherapy was administered, whereas the angiographic end point was determined to be when the arterial flow was reduced to the point that it took 2–5 beats to clear the contrast column.

Intraprocedural Cone-Beam Computed Tomography Acquisition

Cone-beam computed tomography (CT) imaging was performed using a commercial angiographic system (Allura Xper FD20; Philips Healthcare, Best, the Netherlands) with the “XperCT” option, enabling cone-beam CT acquisition and volumetric image reconstruction, immediately before and after chemoembolization. Sequences captured included noncontrast and dual-phase contrast images before and after the procedure as per standard protocol (1). A total of 297 images were acquired over 5 seconds at rate of 60 frames/s via a motorized C-arm in a 240° clockwise arc, under a configuration of fixed 120 kVp. Patients were instructed to perform end expiratory breath holding during the image acquisitions. The reconstruction was performed from 2-dimensional projection images with the Feldkamp back projection technique, resulting in 3-dimensional volumetric images. These images were of size 250 × 250 × 194–mm3 field of view (matrix size, 384 × 384 × 296), with a 0.6-mm3 voxel size.

Convolutional Neural Network Architecture Details

This U-net’s contraction sequence involved 4 sets of 2 convolution layers, with a rectified linear unit after each convulsion, and a max pooling layer after each set. Unique to the U-net is the expansion path. This follows a sequence similar to the contraction path, with up convolutions in place of max pooling, and additional concatenation of features from the contraction path. Given an input, a stack of 3 cone-beam CT slices 3 × 384 × 384, the result is an array of size 2 × 384 × 384. The model predicts the deposition of Lipiodol in the center image of the 3 input slices. The output comes in the form of 2 classes, one mapping the prediction’s confidence of Lipiodol presence and the other is in which Lipiodol is not present. To obtain the final binary segmentation, Lipiodol is said to be wherever the class predicting its presence is greater than the class predicting in which it is not present.

The model’s loss function was soft dice. Soft dice is similar to dice in that the formula is 2 times the sum of the intersected values over the total sum of both images, except that soft dice does not use a binary representation of the images. Instead of both sets of input images being a binary representation of where the Lipiodol is (value of 1) and is not (value of 0), the predicted values formed a heatmap of where the Lipiodol is likely to be. This means that 1 set of input values range from 0 to 1 based on the likelihood of Lipiodol being present in the prediction. The effect of this is that the model is updated based on the prediction’s confidence, not a more ambiguous thresholded value. The benefit of soft dice is that the loss function received the confidence of the model’s prediction, not just a binary classification. In other words, regular dice loses some information by thresholding the model’s prediction, whereas soft dice does no thresholding, and the outcome reflects this when the gradient was updated. The model was built using Python 3.5 and Keras 2.2 (https://keras.io/) running on a TensorFlow backend (Google, https://www.tensorflow.org/). The U-net architecture itself involved 4 downsampling layers followed by 4 upsampling layers. Adam was used as the testing optimizer, and dropout layers were added to prevent overfitting. In total, 1,500 epochs were used. The learning rate was started at 0.0001 and halved every 250 epochs.

Image Augmentation, Preprocessing, and K-Fold Cross Validation

The data set was doubled with the application of 2 types of augmentation, intensity and rotation. For each input in the current study’s set, a duplicate was added with random intensity values shifted between +45 and −45 and with added random rotation of between +60° and −60°. These augmentations were chosen because both the intensity and orientation of the liver vary between cone-beam CT scans (2).

Slices in which no liver was present were removed from the stack. The resulting images were normalized by subtracting the mean and dividing by the standard deviation. In addition to normalization, the minimum and maximum intensities were truncated at the 1st and 99th percentiles of the image’s intensities to remove outliers and potential artifacts (eg, metal/bone or air). This set of images served as inputs to the convolutional neural network.

From the study cohort of 36 patients, the training and testing sets were randomly split into groups of 27 and 9, respectively. The training group served as the input to the neural network for it to learn how to segment Lipiodol, whereas the testing group was used to create predictions and compared with a ground truth in a series of tests. After testing was completed, the patient cohort was shuffled and split back into new training and testing sets for the model to be retrained and re-evaluated. This process of K-fold cross validation was performed over 6 folds, with the evaluation of each new model yielding testing results for all 4 metrics described in the “Results” section.

REFERENCES

  • 1.Loffroy R, Lin M, Yenokyan G, et al. Intraprocedural C-arm dual-phase cone-beam CT: can it be used to predict short-term response to TACE with drug-eluting beads in patients with hepatocellular carcinoma? Radiology 2013; 266:636–648. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Ronneberger O, Fischer P, Brox T. U-net: convolutional networks for biomedical image segmentation. In: Navab N, Hornegger J, Wells W, Frangi A, editors. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. MICCAI 2015. Lecture Notes in Computer Science , vol 9351. Cham, Germany: Springer International Publishing; 2015:234–241. [Google Scholar]

Footnotes

Appendix A can be found by accessing the online version of this article on www.jvir.org and selecting the Supplemental Material tab.

REFERENCES

  • 1.Hu J, Maybody M, Cao G, et al. Lipiodol retention pattern assessed by cone beam computed tomography during conventional transarterial chemoembolization of hepatocellular carcinoma: accuracy and correlation with response. Cancer Imaging 2016; 16:32. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Tacher V, Radaelli A, Lin M, Geschwind JF. How I do it: cone-beam CT during transarterial chemoembolization for liver cancer. Radiology 2015; 274:320–334. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Zhang Y, Yu H. Convolutional neural network based metal artifact reduction in X-ray computed tomography. IEEE Trans Med Imaging 2018; 37:1370–1381. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Gaba RC, Lewandowski RJ, Hickey R, et al. Transcatheter therapy for hepatic malignancy: standardization of terminology and reporting criteria. J Vasc Interv Radiol 2016; 27:457–473. [DOI] [PubMed] [Google Scholar]
  • 5.Tacher V, Lin M, Chao M, et al. Semiautomatic volumetric tumor segmentation for hepatocellular carcinoma: comparison between C-arm cone beam computed tomography and MRI. Acad Radiol 2013; 20: 446–452. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Chen R, Geschwind JF, Wang Z, Tacher V, Lin M. Quantitative assessment of lipiodol deposition after chemoembolization: comparison between cone-beam CT and multidetector CT. J Vasc Interv Radiol 2013; 24:1837–1844. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Wang Z, Lin M, Lesage D, et al. Three-dimensional evaluation of lipiodol retention in HCC after chemoembolization: a quantitative comparison between CBCT and MDCT. Acad Radiol 2014; 21:393–399. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Letzen BS, Malpani R, Miszczuk M, et al. Lipiodol as an intra-procedural imaging biomarker for liver tumor response to transarterial chemoembolization: post-hoc analysis of a prospective clinical trial. Clin Imaging 2021; 78:194–200. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Ronneberger O, Fischer P, Brox T. U-net: convolutional networks for biomedical image segmentation. In: Navab N, Hornegger J, Wells W, Frangi A, editors. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. MICCAI 2015. Lecture Notes in Computer Science , vol 9351. Cham, Germany: Springer International Publishing; 2015. p. 234–241. [Google Scholar]
  • 10.Zahra E, Ali B, Siddique W. Medical image segmentation using a U-Net type of architecture. arXiv preprint arXiv:200505218 2020. [Google Scholar]
  • 11.Dong H, Yang G, Liu F, Mo Y, Guo Y, Automatic brain tumor detection and segmentation using U-Net based fully convolutional networks. Communications in Computer and Information Science. Cham, Germany: Springer International Publishing; 2017:506–517. [Google Scholar]
  • 12.Matsui Y, Horikawa M, Jahangiri Noudeh Y, Kaufman JA, Kolbeck KJ, Farsad K. Baseline tumor Lipiodol uptake after transarterial chemoembolization for hepatocellular carcinoma: identification of a threshold value predicting tumor recurrence. Radiol Oncol 2017; 51:393–400. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Erdi YE, Mawlawi O, Larson SM, et al. Segmentation of lung lesion volume by adaptive positron emission tomography image thresholding. Cancer 1997; 80:2505–2509. [DOI] [PubMed] [Google Scholar]
  • 14.Michetti J, Basarab A, Diemer F, Kouame D. Comparison of an adaptive local thresholding method on CBCT and μCT endodontic images. Phys Med Biol 2017; 63:015020. [DOI] [PubMed] [Google Scholar]
  • 15.Milletari F, Navab N, Ahmadi SA, V-net: fully convolutional neural networks for volumetric medical image segmentation. 2016 Fourth International Conference on 3D Vision (3DV). IEEE; 2016:565–571. [Google Scholar]
  • 16.Zou KH, Warfield SK, Bharatha A, et al. Statistical validation of image segmentation quality based on a spatial overlap index. Acad Radiol 2004; 11:178–189. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Stark S, Wang C, Savic LJ, et al. Automated feature quantification of Lipiodol as imaging biomarker to predict therapeutic efficacy of conventional transarterial chemoembolization of liver cancer. Sci Rep 2020; 10:18026. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Pauwels R, Jacobs R, Singer SR, Mupparapu M. CBCT-based bone quality assessment: are Hounsfield units applicable? Dentomaxillofac Radiol 2015; 44:20140238. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Nagarajappa AK, Dwivedi N, Tiwari R. Artifacts: the downturn of CBCT image. J Int Soc Prev Community Dent 2015; 5:440–445. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Monsky WL, Kim I, Loh S, et al. Semiautomated segmentation for volumetric analysis of intratumoral ethiodol uptake and subsequent tumor necrosis after chemoembolization. AJR Am J Roentgenol 2010; 195: 1220–1230. [DOI] [PubMed] [Google Scholar]
  • 21.Najmi Varzaneh F, Pandey A, Aliyari Ghasabeh M, et al. Prediction of post-TACE necrosis of hepatocellular carcinoma usingvolumetric enhancement on MRI and volumetric oil deposition on CT, with pathological correlation. Eur Radiol 2018; 28:3032–3040. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Kamburoglu K Use of dentomaxillofacial cone beam computed tomography in dentistry. World J Radiol 2015; 7:128–130. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Solbiati M, Passera KM, Goldberg SN, et al. A novel CT to cone-beam CT registration method enables immediate real-time intraprocedural three-dimensional assessment of ablative treatments of liver malignancies. Cardiovasc Intervent Radiol 2018; 41:1049–1057. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Niu K, Yang P, Wu Y, et al. C-arm conebeam CT perfusion imaging in the angiographic suite: a comparison with multidetector CT perfusion imaging. AJNR Am J Neuroradiol 2016; 37:1303–1309. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Ganguly A, Fieselmann A, Marks M, et al. Cerebral CT perfusion using an interventional C-arm imaging system: cerebral blood flow measurements. AJNR Am J Neuroradiol 2011; 32:1525–1531. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Yabo F, Yang L, Yingzi L, et al. Cone-beam computed tomography (CBCT) and CT image registration aided by CBCT-based synthetic CT. ProcSPIE 2020. [Google Scholar]
  • 27.Chen L, Liang X, Shen C, Jiang S, Wang J. Synthetic CT generation from CBCT images via deep learning. Med Phys 2020; 47:1115–1125. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Malpani R, Petty CW, Bhatt N, Staib LH, Chapiro J. Use of artificial intelligence in nononcologic interventional radiology: current state and future directions. Dig Dis Interv 2021; 05:331–337. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Norman B, Pedoia V, Majumdar S. Use of 2D U-Net convolutional neural networks for automated cartilage and meniscus segmentation of knee MR imaging data to determine relaxometry and morphometry. Radiology 2018; 288:177–185. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Miszczuk MA, Chapiro J, Geschwind JH, et al. Lipiodol as an imaging biomarker of tumor response after conventional transarterial chemoembolization: prospective clinical validation in patients mary and secondary liver cancer. Transl Oncol 2020; 13:100742. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.van Breugel JMM, Geschwind JF, Mirpour S, et al. Theranostic application of lipiodol for transarterial chemoembolization in a VX2 rabbit liver tumor model. Theranostics 2019; 9:3674–3686. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Takayasu K, Arii S, Matsuo N, et al. Comparison of CT findings with resected specimens after chemoembolization with iodized oil for hepatocellular carcinoma. AJR Am J Roentgenol 2000; 175: 699–704. [DOI] [PubMed] [Google Scholar]
  • 33.Lin M Accelerating the translation of artificial intelligence from ideas to routine clinical workflow. Acad Radiol 2020; 27:121–122. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES