Skip to main content
PLOS One logoLink to PLOS One
. 2024 Sep 25;19(9):e0308968. doi: 10.1371/journal.pone.0308968

Study on microwave ablation temperature prediction model based on grayscale ultrasound texture and machine learning

Yan Xiong 1,#, Yi Zheng 2,#, Wei Long 1, Yuxin Wang 2, Qin Wang 1, Yi You 1, Yuheng Zhou 1, Jiang Zhong 3, Yunxi Ge 3, Youchen Li 3, Yan Huang 3,*, Zhiyong Zhou 2,4,*
Editor: Carlos Alberto Antunes Viegas5
PMCID: PMC11423965  PMID: 39321182

Abstract

Background

Temperature prediction is crucial in the clinical ablation treatment of liver cancer, as it can be used to estimate the coagulation zone of microwave ablation.

Methods

Experiments were conducted on 83 fresh ex vivo porcine liver tissues at two ablation powers of 15 W and 20 W. Ultrasound grayscale images and temperature data from multiple sampling points were collected. The machine learning method of random forests was used to train the selected texture features, obtaining temperature prediction models for sampling points and the entire ultrasound imaging area. The accuracy of the algorithm was assessed by measuring the area of the hyperechoic area in the porcine liver tissue cross-section and ultrasound grayscale images.

Results

The model exhibited a high degree of accuracy in temperature prediction and the identification of coagulation zone. Within the test sets for the 15 W and 20 W power groups, the average absolute error for temperature prediction was 1.14°C and 4.73°C, respectively. Notably, the model’s accuracy in measuring the area of coagulation was higher than that of traditional ultrasonic grey-scale imaging, with error ratios of 0.402 and 0.182 for the respective power groups. Additionally, the model can filter out texture features with a high correlation to temperature, providing a certain degree of interpretability.

Conclusion

The temperature prediction model proposed in this study can be applied to temperature monitoring and coagulation zone range assessment in microwave ablation.

1. Introduction

As the sixth most common cancer worldwide and the third leading cause of cancer death, primary liver cancer accounted for over 900,000 new cases globally in 2020 alone [1]. Despite a recent decline in the incidence of liver cancer, the poor prognosis and high mortality rate continue to pose a significant medical burden [2], with hepatocellular carcinoma (HCC) accounting for 90% of primary liver malignancies [3] and a five-year overall survival (OS) rate of merely 20% for HCC [4]. Currently, surgical resection, liver transplantation, and ablation therapy are the three main curative treatments for early-stage HCC [4]. Microwave ablation (MWA), as an emerging ablation modality, induces coagulative necrosis and cell death by heating tissue through electromagnetic wave-induced vibration of water molecules [5]. Compared to traditional radiofrequency ablation, MWA’s active heating method is unaffected by impedance, achieving shorter ablation times, higher temperatures, and larger coagulation zone, with less impact due to the heat sink effect [57]. Although MWA has been widely used in liver cancer treatment, difficulty in controlling the energy in the coagulation zone remains a major factor limiting the prognosis of ablation surgery. Insufficient temperature can lead to tumor residue and increased postoperative recurrence risk, while excessive temperature may result in the burning of adjacent normal tissues due to MWA’s high thermal efficiency, leading to a series of complications such as bleeding, bile duct damage, and gastrointestinal perforation [8]. The temperature distribution within the tissue determines the range of tissue damage during ablation, with isotherms between 60°C and 100°C outlining the approximate range for complete tumor necrosis [9], making temperature monitoring in thermal ablation an important aspect of the ablation process.

Magnetic Resonance Imaging (MRI) is considered the "gold standard" among non-invasive thermometry techniques. Thanks to the high temperature dependence of MRI-related tissue parameters, including relaxation parameters (T1, T2) and proton resonance frequency, MRI can perform quantitative temperature monitoring with high spatiotemporal resolution [10]. However, its high cost and the need for ablation methods to be compatible with MRI’s strong magnetic field are major barriers to its clinical application. Computed Tomography (CT) imaging can avoid these limitations. The correlation between CT value attenuation and temperature was demonstrated as early as 1979 [11], but CT thermometry requires multiple scans, exposing patients to higher radiation [12]. Compared to CT and MRI, ultrasound imaging may have a slightly inferior spatial resolution but offers unparalleled convenience. Ultrasonic thermometry is a method for inferring tissue temperature information by analyzing the echo characteristics or texture information changes of ultrasonic waves propagating in tissues. The raw echo signal, also known as the ultrasonic radiofrequency (RF) signal, forms the basis for temperature measurement methods, which primarily include time-shift method [1315], backscatter energy method [1618], Nakagami parameter imaging method [1921], among others. In contrast, temperature measurement methods based on texture information changes in images [22, 23] utilize the grayscale variations and interrelationships of pixels in ultrasonic images to estimate temperature changes [24]. Texture information changes refer to the variations in the distribution and patterns of pixel intensities in ultrasound images, which are associated with the microstructural changes in tissue induced by thermal therapy. By quantitatively analyzing these texture features, we can estimate the temperature distribution within the tissue. Due to the convenience of grayscale image acquisition, this method exhibits broader clinical application prospects compared to other thermometry techniques. Temperature changes are reflected in texture features as variations in multiple rather than single texture parameters. To assess the importance of these parameters in temperature prediction, we introduced the random forest method from machine learning to analyze these parameters through weighted voting.

This study proposes an ultrasound temperature prediction machine learning method based on radiomics, which involves first-order statistical analysis and higher-order texture feature extraction of ultrasound images around multiple thermometry points. The random forest algorithm is used to select the ten most important texture features and establish a model correlating these features with temperature, aiming to predict the temperature within a specific ultrasound imaging area.

2. Experimental setup and data collection

2.1 Experimental equipment

Ultrasound Equipment: The Vinno G86 medical ultrasound machine, manufactured in Suzhou, China, employs the X6-6L model linear array transducer for signal transmission and reception. For grayscale signal collection, the parameters are as follows: line density set to 128, the ultrasound probe frequency of 12 MHz, and a frame rate of 15 frames per second. The system operates in fundamental imaging mode with spatial compounding deactivated.

Ablation Equipment: The Ruibo microwave ablation therapy instrument, manufactured in Nanjing, China, is model MWD-A. It operates at a frequency of 2450 MHz and has a maximum output power of 100 W. The sterile disposable microwave ablation needle is model WGP-Z03, with a diameter of 1.5mm and a length of 100mm.

Thermometer: The Keysight 34970A, used for temperature measurements, samples at a frequency of 2 Hz through channels 101 to 107 using T-type thermocouple sensor wires. To ensure accurate temperature monitoring during the ablation process, a cold circulation system with 0.9% saline solution was employed.

2.2 Biological materials

Fresh porcine livers were trimmed to conform to the dimensions of the mold. They were then submerged in a water bath maintained at a temperature of 37°C, with a water depth of 8 centimeters. This procedure was employed to achieve a uniform heating of the liver samples, ensuring that their temperature exceeded 36°C. The mold was utilized to secure the position of the porcine liver during the heating process, ensuring consistency in the experimental conditions.

2.3 Temperature measurement mold and thermocouple needles

Due to the high density of biological materials, hollow needles (length: 97mm, diameter: 1.45mm) were used as the outer shell to guide the embedded thermocouples accurately to the predetermined positions, as shown in Fig 1A. To ensure that all temperature measurement points and the ultrasound imaging plane for image collection were in the same plane during the ablation process, a mold was designed to fix the temperature measurement needles and the ablation needle, as shown in Fig 1B. The mold, with specifications of 20 × 18 × 10 cm, provided puncture holes for simultaneous access of one ablation needle and seven temperature measurement needles. The relative positions of the temperature measurement points and the ultrasound scanning plane are shown in Fig 1C. Based on pre-experiments on the fusiform coagulation zone of porcine liver, the positions of the temperature measurement points were set, with all temperature measurement holes located below the ablation needle. Point 1 was located to the left of the tip of the ablation needle, and the rest were evenly distributed near the ablation needle to ensure more effective acquisition of temperature changes at different positions within the entire coagulation zone.

Fig 1. Probe assembly, mold design, and positioning schematic.

Fig 1

(A) Temperature probe assembly details: a. Thermocouple wire—Responsible for transmitting temperature signals. b. Hollow needle tip—Serves as the protective and guiding part for the sensor wire. c. Assembled temperature probe—Demonstrates the assembly of the sensor wire and needle tip. (B) Mold design for temperature probes and ablation needle: The diagram shows the expected layout of the temperature probes and ablation needle within the mold, with the ablation needle marked in red and the temperature probes marked in blue. (C) Expected relative positions of the temperature measurement points: This details the precise layout of the temperature measurement points relative to the ablation needle. The ablation needle is indicated by blue arrows (tip to the left), the temperature measurement points are represented by red dots (indexed at the bottom), and black lines denote the expected spatial distances between the measurement points.

2.4 Experimental procedure

Fresh porcine livers were cut into tissue blocks with a volume of 6 × 6 × 4 cm to fit the mold size and placed in a constant temperature water bath at 37°C, along with the mold, to heat the porcine liver to above 36°C. The ultrasound transducer was closely attached to the liver surface, and a region with uniform grayscale was selected. The ablation needle and temperature measurement needles were inserted along the mold. The microwave ablation instrument parameters were set (heating power P = 15 W, 20 W; heating time t = 180 s). The positions of each temperature measurement needle were marked and recorded in the ultrasound images. After the ablation, ultrasound recorded videos and multi-point temperature data were obtained, and photos of the liver tissue sections at the end of the ablation were taken. A total of 42 and 41 ex vivo porcine liver microwave ablation experiments were conducted at the two power levels, respectively. Fig 2 shows the actual experimental setup.

Fig 2. Experimental setup photo.

Fig 2

3. Dataset and methodology

Fig 3 illustrates a series of processing steps applied to the raw data. Initially, keyframes with corresponding temperature data are extracted from the ultrasound ablation videos. Based on these keyframes, the regions of interest (ROI) containing the temperature measurement points are accurately segmented. These temperature-ROI sample pairs constitute the input portion of the temperature prediction model. Subsequently, imaging biomarker features are extracted from these ROIs, encompassing first-order statistical features and higher-order texture features. These features provide rich image information, facilitating the model’s capture of temperature variations. Ultimately, leveraging these features, temperature at the measurement points is predicted, and thermal maps of the ultrasound imaging region are generated based on these prediction data.

Fig 3. Ultrasound temperature prediction method.

Fig 3

This figure outlines the methodological framework for extracting keyframes from ultrasound videos with corresponding temperature data. It then details the precise segmentation of regions of interest (ROIs) around temperature measurement points. The figure further illustrates the process of extracting imaging biomarker features from these ROIs, including comprehensive first-order statistical features and higher-order texture features. These features play a crucial role in capturing the details of temperature changes within the ablation zone and are subsequently used to generate a temperature estimation prediction model.

3.1 Preprocessing of experimental data and dataset construction

3.1.1 Preprocessing of temperature data

Given the multiple temperature measurement channels involved in the experiment, a varying preheating time is required for the initiation of temperature data collection by the thermometers. To synchronize data collection between the thermometers and the ultrasound machine, and to allow sufficient reaction time for the activation of the ultrasound and ablation devices, we first initiated the thermometers. Temperature data collection commenced at the 15th second after the preheating period, simultaneously with the start of ultrasound device video recording and the activation of the ablation device. The temperature data corresponding to the ablation process (from 15 seconds to 195 seconds) was retained.

3.1.2 Preprocessing of ultrasound images

  1. Frame extraction from ultrasound videos. The ultrasound videos captured during the experiment had a frame rate of 12 frames per second. To synchronize the temperature and image data, we extracted the first and sixth frames of each second from the video, corresponding to the temperature data collection frequency of twice per second.

  2. Segmentation of ROI at the temperature measurement points. Based on the coordinates of the temperature measurement points marked in the ultrasound images before ablation, a square region of 64 × 64 pixels centered on the marked points was designated as ROI. Each extracted frame was segmented accordingly, as shown in Fig 4. The temperature data and ROI were matched according to the numbering recorded during the data collection process, ultimately forming temperature-ROI sample pairs.

Fig 4. Ultrasound image showing the ablation needle and temperature measurement ROIs.

Fig 4

This pre-ablation image captures the initial setup, with the ablation needle positioned accurately and the ROIs for temperature measurement clearly marked. Each 64 × 64 pixel square ROI is centered on a temperature measurement point, indicating the areas where temperature changes will be monitored throughout the ablation process.

3.2 Extraction of radiomic features and construction of the temperature prediction model

In the analysis utilizing the Intelligent Medical Imaging Assistant System [25], we performed extensive texture feature extraction on the regions of interest (ROIs) around the temperature measurement points using radiomics methods. These features included not only first-order statistical features (n = 19) that describe the grayscale distribution of the image, but also 2D shape-based features (n = 10) that help quantify the geometric properties of the image. Additionally, we extracted various higher-order texture features that describe the relationships between pixels, including the Gray Level Co-occurrence Matrix (n = 24), Gray Level Run Length Matrix (n = 16), Gray Level Size Zone Matrix (n = 16), Neighbouring Gray Tone Difference Matrix (n = 5), and Gray Level Dependence Matrix (n = 14). These features collectively reflect the complexity of the image texture and provide critical information for predicting the temperature distribution within the coagulation zone during microwave ablation. The radiomic features were normalized and subjected to feature selection. The selected features were then used as inputs to train the ultrasound image temperature prediction model, with temperature as the output.

In this study, feature importance was assessed using the Gini importance index [26] from the random forest algorithm [27]. The Gini importance index measures a feature’s ability to reduce impurity at each split point within the random forest. A high Gini coefficient indicates that the feature provides more information during classification or regression, thus contributing more significantly to the model’s predictive power. By calculating the Gini coefficient for each feature, we ranked their importance accordingly. (Fig 5) and selected the top 10 texture features in terms of importance. We believe these features are highly correlated with temperature. These features were then used to construct the temperature prediction model, which was fitted with temperature data through regression analysis. The model optimization was based on performance on the validation set to ensure optimal performance in temperature prediction.

Fig 5. Ranking of features by importance in the 15 W and 20 W ablation groups.

Fig 5

This figure displays the importance ranking of temperature prediction features identified by the random forest algorithm, based on their contribution to predicting temperature changes. It highlights the features with the greatest predictive value during the microwave ablation process.

The experiment collected and organized data from 42 cases with an ablation power of 15W and 41 cases with 20W. Other experimental parameters were consistent across all cases. The temperature data collection frequency was 2Hz, with seven temperature measurement points, and the ultrasound video frame rate was 12Hz, with a total duration of 3 minutes. Key frames were extracted from the videos based on the fixed preheating time and temperature data collection frequency, resulting in 360 frames of ultrasound image data. The temperature at the seven measurement points for each frame was obtained, forming a total of 2520 temperature-ultrasound image sample pairs for each case. The samples were randomly divided into training, validation, and test sets in a 4:1:1 ratio. By training the ultrasound temperature prediction algorithm on classified samples, we have successfully established a temperature prediction model. This model can be utilized for subsequent temperature forecasting tasks.

4. Experimental results

4.1 Evaluation metrics for the prediction model

The temperature prediction model is used for single-point temperature prediction and thermal map prediction. In single-point temperature prediction, the model predicts the temperature at the center of the region of interest (ROI) corresponding to the measured point and compares it with the actual temperature obtained by the thermometer. In thermal map prediction, the model predicts the temperature of each pixel in the plane of the microwave ablation center area at the end of ablation, then stitches all the pixels together to present a temperature distribution map. In this study, the evaluation metrics for single-point temperature prediction are mean absolute error (MAE) and root mean square error (RMSE), while the evaluation metric for thermal map prediction is the error ratio.

  • ①Mean Absolute Error (MAE) for single-point temperature prediction:

MAE(MeanAbsoluteError)=1ni=1n|yiy^i|

Where n is the total number of temperature prediction samples, yi is the actual temperature value of sample i, and y^i is the predicted temperature value of sample i.

  • ②Root Mean Square Error (RMSE) for single-point temperature prediction:

RMSE(RootMeanSquareError)=i=1n(yiy^i)2n

where n is the total number of temperature prediction samples, yi is the actual temperature value of sample i, and y^i is the predicted temperature value of sample i.

  • ③Error Ratio (ER) at the end of ablation for pixel area threshold and clinical observation area:

This study precisely calculated the clinical observation area Sv delineated by the hyperechoic regions in ultrasound images after ablation, as well as the actual coagulation zone area Sr outlined by the brownish-yellow parenchymal region in porcine liver post-ablation. Accurate quantitative analyses of these two evaluation metrics were achieved by integrating ultrasound images (S3 Fig) with gross specimen photographs (S1 Fig). Additionally, temperature distribution heat map predictions (S2 Fig) were made for specified regions based on the ablation center in ultrasound images. First, pixel blocks in the target area were extracted, and a regression model was constructed using the radiomic features of the extracted pixel blocks to predict the temperature. Then, based on a set temperature threshold, areas in the heat map with temperatures above the threshold were segmented and their areas were calculated to obtain the pixel area Sp. By separately comparing Sv and Sp with Sr, it is possible to calculate the error ratio (ER) of two different prediction methods. The closer ER is to 0, the smaller the prediction error of the coagulation zone area for the respective prediction method.

ER(ErrorRatio)=|xSr|Sr

In this equation, x represents the measured area of the ablation zone (Sv) or the predicted area (Sp), as indicated in the corresponding analysis.

4.2 Analysis of temperature prediction results

In the experiment with an ablation power of 15 W, the temperature prediction of the validation set showed that the model’s MAE was 2.84°C, and the RMSE was 4.74°C. The temperature prediction of the test set further demonstrated the accuracy of the model, with an MAE reaching 1.14°C. Three samples involving a larger temperature range were selected from the experimental dataset for the analysis of the entire ablation process temperature prediction (Fig 6). The MAE of temperature measuring point 1 was 1.71°C, that of temperature measuring point 2 was 2.08°C, and that of temperature measuring point 3 was 2.14°C. The experimental prediction results indicate that for the specific temperature measuring points mentioned above, the temperature prediction shows a close trend. However, there is a phenomenon where the prediction error between the predicted temperature and the actual temperature is larger in the early and late stages of ablation.

Fig 6. Comparison of predicted and actual temperatures at 15W ablation (data presented in S1 Table).

Fig 6

In the experiment with an ablation power of 20 W, the MAE and RMSE of the validation set were 5.22°C and 7.61°C, respectively, showing an increase in error compared to the 15 W power group, which may be related to the overall increase in the temperature range involved. The MAE of the test set decreased to 4.73°C, indicating that the model still possesses good predictive accuracy at this power level. Three samples involving different temperature ranges were selected from the experimental dataset for the analysis of the entire ablation process temperature prediction (Fig 7). The MAE of measuring point 1 was 2.44°C, the MAE of measuring point 2 was 1.44°C, and the MAE of measuring point 3 was 0.89°C. The experimental prediction results indicate that for measuring point 1 in the 20 W power group, the overall prediction error is higher due to the larger range of prediction, as the maximum temperature reached over 75°C. In contrast, measuring points 2 and 3 had lower maximum temperatures, resulting in smaller prediction errors.

Fig 7. Comparison of predicted and actual temperatures at 20W ablation (data presented in S2 Table).

Fig 7

4.3 Analysis of predicted results in the coagulation zone

Representative ex vivo porcine liver ultrasound grayscale images, machine learning-predicted heatmaps, threshold heatmaps, and tissue section images collected immediately after MWA are presented in Fig 8. These images are divided into two groups based on the power used: 15W (Fig 8A–8D) and 20W (Fig 8E–8H). The threshold heatmaps display pixels with temperatures above 54°C and calculate their area. In clinical ablation procedures, temperatures above 54°C can ensure complete necrosis of tumor cells [5] and are close to the temperature at the ablation boundary. After ablation, the liver tissue is cut along the microwave ablation needle tract, and the clearly demarcated brownish-yellow area around the needle tract center is identified as the ablation coagulation range (Fig 8D and 8H). The coagulation area is outlined and measured using ImageJ software to obtain the actual coagulation area, which serves as the gold standard.

Fig 8. Ultrasound images and corresponding predicted heat maps at the end of ablation.

Fig 8

This figure includes ex vivo porcine liver grayscale ultrasound images, machine learning-predicted thermographs, threshold thermographs, and tissue cross-sections. All images are categorized by ablation power, with the 15 W and 20 W groups displayed side by side for comparison.

During ablation, an elliptical high-echo hyperechoic area centered around the ablation needle gradually expands on the ultrasound image, caused by microbubbles formed due to the temperature increase. The major and half the minor diameters (a, b) of the hyperechoic area’s largest cross-section are measured immediately after ablation (Fig 9), and the area of the cavitation range is calculated using the formula for an ellipse: Sv=12×a×b×π

Fig 9. Ultrasound hyperechoic area measurement.

Fig 9

In our study, Tables 1 and 2 are specifically designed to evaluate the accuracy of machine learning (ML) methods in predicting the coagulation zone. These tables compare the area of the coagulation zone predicted by ML Sp with the gold standard area Sr obtained using ImageJ software, and also include the area of the hyperechoic area in ultrasound grayscale images Sv. We selected samples from the 15W and 20W ablation groups and predicted the coagulation zone using the ML method based on the 54°C isotherm. This threshold is close to the temperature at the ablation boundary and represents the minimum temperature required for protein denaturation, as determined in previous studies. From the data in Tables 1 and 2, we observe that the mean error ratios for the ML measurements in the 15W and 20W ablation groups are 0.159 and 0.122, respectively. These values are significantly lower than the mean error ratios for the hyperechoic area measurements in ultrasound grayscale images, which are 0.402 and 0.182, respectively. This demonstrates the superior accuracy of the ML method in predicting the coagulation zone.

Table 1. Prediction results of the coagulation area (P = 15W, t = 180s).

Gold Standard (mm2) Measured Hyperechoic Area (mm2) Predicted by ML (mm2) Error Ratio
Hyperechoic Area ML
192.969 258.423 133.690 0.339 0.307
153.100 221.558 146.042 0.447 0.046
176.083 233.870 144.782 0.328 0.178
152.061 218.191 178.112 0.435 0.171
169.047 246.556 153.136 0.459 0.094

Table 2. Prediction results of the coagulation area (P = 20W, t = 180s).

Gold Standard (mm2) Measured Hyperechoic Area (mm2) Predicted by ML (mm2) Error Ratio
Hyperechoic Area ML
193.357 236.522 188.719 0.223 0.024
261.281 308.494 205.808 0.181 0.212
256.924 301.556 207.278 0.174 0.193
241.527 281.911 217.388 0.167 0.100
264.442 307.946 242.652 0.165 0.082

4.4 Model error and correlation analysis of texture features with temperature

Through feature selection on the experimental data of 42 cases with an ablation power of 15 W, the top three features ranked by importance are Gray Level Co-occurrence Matrix (GLCM) Cluster Prominence, First Order Variance, and First Order Mean Absolute Deviation. Among these, the Cluster Prominence feature is the most important in the 15 W random forest model, with an importance value of 1.2692. The importance rankings from the random forest model indicate the correlation between these three features and temperature, hence they were chosen as representative features and subjected to regression analysis with the corresponding temperature data. We selected a case from the 15 W dataset with good temperature prediction results. Fig 10 shows the fitting results of the three representative features with temperature in this dataset, with each scatter plot containing 2520 data points. The linear correlation coefficients between Cluster Prominence, Variance, and Mean Absolute Deviation with temperature are -0.8742, -0.7573, and -0.6760, respectively.

Fig 10. Scatter plots of representative features with temperature in the 15 W ablation group.

Fig 10

(a) Cluster Prominence; (b) Variance; (c) Mean Absolute Deviation.

Through feature selection on the experimental data of 41 cases with an ablation power of 20 W, the top three features ranked by importance are First Order Variance, First Order Mean Absolute Deviation, and First Order Minimum. Among these, Variance is the most important feature in the 20 W random forest model, with an importance value of 2.0368. A case from the 20 W dataset with good temperature prediction results was selected, and these three representative features were subjected to regression analysis with temperature. Fig 11 shows the fitting results of the three features with temperature in this dataset, with each scatter plot containing 2520 data points. The linear correlation coefficients between Variance, Mean Absolute Deviation, and Minimum with temperature are -0.9016, -0.9061, and 0.7535, respectively.

Fig 11. Scatter plots of representative features with temperature in the 20 W ablation group.

Fig 11

(a) Variance; (b) Mean Absolute Deviation; (c) Minimum Feature Value.

5. Discussion

To address the challenge of determining the coagulation zone boundaries during clinical ablation procedures, we proposed a temperature monitoring method under ultrasound imaging based on supervised machine learning. The results indicate that this method can more effectively assess the coagulation zone compared to the commonly used clinical practice of evaluating ablation status based on grayscale ultrasound images. Our study addresses several important issues.

Compared to RF signals, grayscale signals are more convenient to acquire. RF signal acquisition typically requires research-grade ultrasound scanners, whereas grayscale signals can be collected using clinically used ultrasound devices. Texture features are obtained from grayscale signals, and many studies have shown a correlation between texture parameters and temperature [22, 28, 29]. Alvarenga et al. [22] analyzed temperature changes using certain parameters (entropy and correlation) of the gray level co-occurrence matrix (GLCM) by heating tissue phantoms in a water bath, showing that these parameters are highly sensitive at low temperatures. Yang et al. [28] studied the relationship between texture features and tissue temperature during microwave ablation, finding that GLCM parameters (contrast, angular second moment, inverse difference moment, and correlation) showed high correlation over a broader temperature range. Finally, Wang et al. [29] expanded this field by investigating texture features based on gray level histograms, GLCM, and gray level gradient co-occurrence matrix in heated porcine liver samples, demonstrating that the correlation coefficients of five feature parameters exceeded 0.9 in the temperature range of 20°C to 60°C. Our study comprehensively evaluated texture parameters by screening 104 feature values, obtaining features with high correlation to temperature for model testing. Not only can the trends of feature values with temperature changes be visually observed through scatter plots, but the selected texture features also have interpretability. In our study, although the important features identified for temperature prediction varied across different power groups, we observed a clear consistent trend: as temperature increased, the grayscale values of the ablated tissue also increased, leading to greater image uniformity. Notably, features such as GLCM cluster prominence, variance, mean absolute deviation, and minimum value exhibited high importance in both ablation groups, playing crucial roles in temperature prediction. Correlation analysis of these features with temperature revealed significant patterns: the minimum value was positively correlated with temperature, indicating a general increase in grayscale values within the region of interest (ROI) as temperature rose. Conversely, GLCM cluster prominence, variance, and mean absolute deviation were negatively correlated with temperature. As measures of skewness and asymmetry, these features decreased with rising temperature, reflecting increased uniformity in grayscale images due to tissue coagulation and bubble formation. Despite potential variations in temperature-related features identified by different studies due to differences in experimental conditions and tissue characteristics, we believe the observed pattern of grayscale image changes with temperature is universal. This conclusion holds clinical significance across various contexts, offering a reliable method for predicting and assessing temperature changes in coagulation zone.

For machine learning, achieving ideal results often requires a large amount of data for training. Traditional ablation experiments usually use a single temperature-measuring needle parallel to the ablation needle [13, 30], collecting data from only one position per ablation. In contrast, our study innovatively used a custom mold, allowing the thermocouples to be evenly distributed in a plane perpendicular to the ablation needle. A single ablation experiment can simultaneously collect temperature data from seven positions, improving data collection efficiency and making sampling in different positions of the ablation area more representative.

Our experiment achieved good results in temperature prediction, with mean absolute errors within 5°C, and good prediction results for temperatures above the critical temperature. Additionally, by predicting the temperature of all pixels in the selected coagulation zone, we obtained the corresponding heatmaps, which better illustrate the spatial temperature distribution in the entire ultrasound imaging area and closely match the gross gold standard in predicted area. In the experimental prediction of single-point temperatures, we observed a deviation between predicted and actual values, with errors primarily concentrated at the initiation and termination phases of ablation. This deviation may stem from the rapid temperature changes during the initial phase of ablation, leading to a sharp increase in the number of bubbles, as well as reduced image variation due to decreased bubble aggregation at the end of ablation, partially masking texture information of underlying tissues. Additionally, as ablation power and temperature increase, prediction errors correspondingly escalate. During the experiment, we observed that the hyperechoic region was generally larger than the coagulation zone, and the gap between the two gradually decreased as ablation time was extended. This observation is consistent with the findings of Uehara T et al. [31], who introduced the concept of the hyperechoic margin, which aligns with what we observed. Additionally, we noted that the machine learning (ML) method’s predictions of coagulation zone areas were typically smaller than the gold standard data delineated from ex vivo tissues. We speculate that this discrepancy may arise from the ML method’s limitations in recognizing the coagulation zone’s edges during the image segmentation process. The ML method predicts the area by summing the temperatures of all pixel points above 54°C during the image segmentation process. While this threshold is close to the temperature at the ablation boundary, the inherent error in temperature prediction may result in an underestimation of boundary temperatures. Especially, the scarcity of temperature data around 54°C during data collection might further affect the ML method’s predictive accuracy for this critical temperature range. In the temperature prediction heat maps, we noted abnormally elevated temperatures in certain areas, which, in reality, were at a distance from the coagulation zone. This could be attributed to bubble diffusion in vessels and fascia, potentially causing misinterpretation of ablation efficacy. Thus, there remains potential for improvement in the accuracy of our temperature prediction and heat map models.

Our experiment has some limitations. Firstly, using thermocouples as the gold standard for temperature measurement means that even if the prediction results are close to the thermocouple measurements, they only represent the temperature at the corresponding spatial position. Evaluating the temperature of the entire coagulation zone may require magnetic resonance imaging. Furthermore, there exists a disparity between the ablation method employed in our experiments and the actual conditions of microwave ablation for liver cancer in clinical practice. In clinical settings, microwave ablation therapy for liver cancer typically utilizes power ranging from 60 W to 100 W for heating, a range considered suitable for ablation [32]. However, in our experimental research, due to the limited thickness of porcine liver samples and the absence of heat dissipation effects from blood flow in the in vivo environment, we adjusted the ablation strategy based on the requirements of data collection from our experiments to ensure a more extensive temperature distribution and prevent the diffusion of ablation edges to the liver capsule surface. This adjustment involved lowering the ablation power and extending the heating time to 180 seconds to achieve precise control over the ablation effect. Lastly, our experiments were conducted on ex vivo porcine liver samples, whereas in actual ablation surgeries for clinical patients, considerations must be made for the effects of blood flow and respiratory motion, which are factors that cannot be overlooked. These influencing factors are also the primary reasons why various temperature measurement techniques have not yet been successfully applied in clinical ablation surgeries.

6. Conclusion

This study proposed a predictive model for tissue temperature during microwave ablation therapy. We used the random forest method to select and train ultrasound grayscale texture features to evaluate tissue temperature, providing guidance for monitoring the coagulation zone of liver tumor microwave ablation.

Supporting information

S1 Table. Predicted and measured temperatures for the 15 W power group.

(XLSX)

pone.0308968.s001.xlsx (41.2KB, xlsx)
S2 Table. Predicted and measured temperatures for the 20 W power group.

(XLSX)

pone.0308968.s002.xlsx (40.4KB, xlsx)
S1 Fig. Cross-sectional photos of liver post-ablation for 15 W and 20 W power groups.

(PDF)

pone.0308968.s003.pdf (962KB, pdf)
S2 Fig. Thermal maps for 15 W and 20 W power groups: Machine learning predictions and 54°C threshold highlights.

(PDF)

pone.0308968.s004.pdf (686.9KB, pdf)
S3 Fig. Ultrasound greyscale images at the end of ablation for 15 W and 20 W power groups.

(PDF)

pone.0308968.s005.pdf (674.7KB, pdf)

Data Availability

All relevant data are in the paper and its Supporting Information files.

Funding Statement

This work was supported partly by National Natural Science Foundation of China (62271480), Youth Innovation Promotion Association CAS (2021324), Jiangsu Key Technology Research Development Program (BE2021612), Science and Technology Development Project of Suzhou (SYG202321), Postgraduate Research and Practice Innovation Program of Jiangsu Province (KYCX23_2093). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

References

  • 1.Sung H, Ferlay J, Siegel RL, Laversanne M, Soerjomataram I, Jemal A, et al. Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA-Cancer J Clin. 2021;71(3):209–49. doi: 10.3322/caac.21660 WOS:000614520100001. [DOI] [PubMed] [Google Scholar]
  • 2.Rumgay H, Arnold M, Ferlay J, Lesi O, Cabasag CJ, Vignat J, et al. Global burden of primary liver cancer in 2020 and predictions to 2040. J Hepatol. 2022;77(6):1598–606. doi: 10.1016/j.jhep.2022.08.021 WOS:000928043800015. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Lafaro KJ, Demirjian AN, Pawlik TM. Epidemiology of Hepatocellular Carcinoma. Surg Oncol Clin N Am. 2015;24(1):1–+. doi: 10.1016/j.soc.2014.09.001 WOS:000345810300003. [DOI] [PubMed] [Google Scholar]
  • 4.Grandhi MS, Kim AK, Ronnekleiv-Kelly SM, Kamel IR, Ghasebeh MA, Pawlik TM. Hepatocellular carcinoma: From diagnosis to treatment. Surg Oncol-Oxf. 2016;25(2):74–85. doi: 10.1016/j.suronc.2016.03.002 WOS:000377935600002. [DOI] [PubMed] [Google Scholar]
  • 5.Liang P, Yu J, Lu MD, Dong BW, Yu XL, Zhou XD, et al. Practice guidelines for ultrasound-guided percutaneous microwave ablation for hepatic malignancy. World J Gastroenterol. 2013;19(33):5430–8. doi: 10.3748/wjg.v19.i33.5430 WOS:000323920800004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Wright AS, Sampson LA, Warner TF, Mahvi DM, Lee FT. Radiofrequency versus microwave ablation in a hepatic porcine model. Radiology. 2005;236(1):132–9. doi: 10.1148/radiol.2361031249 WOS:000229905300017. [DOI] [PubMed] [Google Scholar]
  • 7.Izzo F, Granata V, Grassi R, Fusco R, Palaia R, Delrio P, et al. Radiofrequency Ablation and Microwave Ablation in Liver Tumors: An Update. Oncologist. 2019;24(10):E990–E1005. doi: 10.1634/theoncologist.2018-0337 WOS:000490365300001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.De Muzio F, Cutolo C, Dell’Aversana F, Grassi F, Ravo L, Ferrante M, et al. Complications after Thermal Ablation of Hepatocellular Carcinoma and Liver Metastases: Imaging Findings. Diagnostics. 2022;12(5):15. doi: 10.3390/diagnostics12051151 WOS:000801940100001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Saccomandi P, Schena E, Massaroni C, Fong Y, Grasso RF, Giurazza F, et al. Temperature monitoring during microwave ablation in ex vivo porcine livers. Ejso. 2015;41(12):1699–705. doi: 10.1016/j.ejso.2015.08.171 WOS:000366787300022. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Lewis MA, Staruch RM, Chopra R. Thermometry and ablation monitoring with ultrasound. Int J Hyperthermia. 2015;31(2):163–81. doi: 10.3109/02656736.2015.1009180 WOS:000353167300008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Bydder GM, Kreel L. The temperature dependence of computed tomography attenuation values. Journal of computer assisted tomography. 1979;3(4):506–10. doi: 10.1097/00004728-197908000-00013 MEDLINE:457964. [DOI] [PubMed] [Google Scholar]
  • 12.Tsapaki V, Tsalafoutas IA, Triantopoulou C, Kolliakou E, Maniatis P, Papailiou J. Radiation dose in repeated CT guided radiofrequency ablations. Phys Medica. 2014;30(1):128–31. doi: 10.1016/j.ejmp.2013.04.002 WOS:000330149000018. [DOI] [PubMed] [Google Scholar]
  • 13.Liu YD, Li Q, Zhou ZH, Yeah YW, Chang CC, Lee CY, et al. Adaptive ultrasound temperature imaging for monitoring radiofrequency ablation. PLoS One. 2017;12(8):15. doi: 10.1371/journal.pone.0182457 WOS:000408365000018. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Huang CW, Lien DH, Chen BT, Shieh J, Tsui PH, Chen CS, et al. Ultrasound thermal mapping based on a hybrid method combining cross-correlation and zero-crossing tracking. J Acoust Soc Am. 2013;134(2):1530–40. doi: 10.1121/1.4812874 WOS:000322739300011. [DOI] [PubMed] [Google Scholar]
  • 15.Park S, Hwang J, Park JE, Ahn YC, Kang HW. Application of Ultrasound Thermal Imaging for Monitoring Laser Ablation in Ex Vivo Cardiac Tissue. Lasers Surg Med. 2020;52(3):218–27. doi: 10.1002/lsm.23157 WOS:000485451600001. [DOI] [PubMed] [Google Scholar]
  • 16.Tsui PH, Chien YT, Liu HL, Shu YC, Chen WS. Using ultrasound CBE imaging without echo shift compensation for temperature estimation. Ultrasonics. 2012;52(7):925–35. doi: 10.1016/j.ultras.2012.03.001 WOS:000306129200017. [DOI] [PubMed] [Google Scholar]
  • 17.Xia JJ, Li Q, Liu HL, Chen WS, Tsui PH. An Approach for the Visualization of Temperature Distribution in Tissues According to Changes in Ultrasonic Backscattered Energy. Comput Math Method Med. 2013;2013:10. doi: 10.1155/2013/682827 WOS:000326517900001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Wang CY, Zhou ZH, Chang YH, Ho MC, Lu CM, Wu CH, et al. Ultrasound single-phase CBE imaging for monitoring radiofrequency ablation of the liver tumor: A preliminary clinical validation. Front Oncol. 2022;12:11. doi: 10.3389/fonc.2022.894246 WOS:000837144900001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Zhang SY, Zhou FY, Wan MX, Wei M, Fu QY, Wang X, et al. Feasibility of using Nakagami distribution in evaluating the formation of ultrasound-induced thermal lesions. J Acoust Soc Am. 2012;131(6):4836–44. doi: 10.1121/1.4711005 WOS:000309133500071. [DOI] [PubMed] [Google Scholar]
  • 20.Wang CY, Geng XN, Yeh TS, Liu HL, Tsui PH. Monitoring radiofrequency ablation with ultrasound Nakagami imaging. Med Phys. 2013;40(7):12. doi: 10.1118/1.4808115 WOS:000321272200052. [DOI] [PubMed] [Google Scholar]
  • 21.Zhou ZH, Wu SC, Wang CY, Ma HY, Lin CC, Tsui PH. Monitoring Radiofrequency Ablation Using Real-Time Ultrasound Nakagami Imaging Combined with Frequency and Temporal Compounding Techniques. PLoS One. 2015;10(2):15. doi: 10.1371/journal.pone.0118030 WOS:000349444900252. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Alvarenga AV, Teixeira CA, Ruano MG, Pereira WCA. Influence of temperature variations on the entropy and correlation of the Grey-Level Co-occurrence Matrix from B-Mode images. Ultrasonics. 2010;50(2):290–3. doi: 10.1016/j.ultras.2009.09.002 WOS:000271779500032. [DOI] [PubMed] [Google Scholar]
  • 23.Teixeira CA, Alvarenga AV, Cortela G, von Krüger MA, Pereira WCA. Feasibility of non-invasive temperature estimation by the assessment of the average gray-level content of B-mode images. Ultrasonics. 2014;54(6):1692–702. doi: 10.1016/j.ultras.2014.02.021 WOS:000338610100034. [DOI] [PubMed] [Google Scholar]
  • 24.Chen M. Ultrasound-based noninvasive temperature estimation in radiofrequency ablation of tumors [Academic Degree of Master]: Southeast University; 2022. (in Chinese).
  • 25.Zhou Z, Qian X, Hu J, Zhu J, Geng C, Dai Y. A new machine learning based user-friendly software platform for automatic radiomics modeling and analysis. Annual International Conference of the IEEE Engineering in Medicine and Biology Society IEEE Engineering in Medicine and Biology Society Annual International Conference. 2021;2021:2810–4. doi: 10.1109/EMBC46164.2021.9630472 MEDLINE:34891833. [DOI] [PubMed] [Google Scholar]
  • 26.Boulesteix AL, Bender A, Bermejo JL, Strobl C. Random forest Gini importance favours SNPs with large minor allele frequency: impact, sources and recommendations. Brief Bioinform. 2012;13(3):292–304. doi: 10.1093/bib/bbr053 WOS:000304536200003. [DOI] [PubMed] [Google Scholar]
  • 27.Zhou Z-H. Machine Learning. 1st ed. 2021 ed: Springer Nature; Singapore; 2021. August 21, 2021. [Google Scholar]
  • 28.Yang CL, Zhu H, Wu SC, Bai YP, Gao HJ. Correlations Between B-Mode Ultrasonic Image Texture Features and Tissue Temperature in Microwave Ablation. J Ultrasound Med. 2010;29(12):1787–99. doi: 10.7863/jum.2010.29.12.1787 WOS:000285702700015. [DOI] [PubMed] [Google Scholar]
  • 29.Wang XL, Sheng L. Correlations between B-mode ultrasound image texture features and tissue temperatures in hyperthermia. PLoS One. 2022;17(10):13. doi: 10.1371/journal.pone.0266446 WOS:000925060900004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Giurazza F, Massaroni C, Silvestri S, Zobel BB, Schena E. Preliminary analysis of ultrasound elastography imaging-based thermometry on non-perfused ex vivo swine liver. Journal of ultrasound. 2020;23(1):69–75. doi: 10.1007/s40477-019-00407-z MEDLINE:31541360. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Uehara T, Hirooka M, Kisaka Y, Abe M, Hiasa Y, Onji M. Usefulness of the hyperechoic rim for assessing the therapeutic efficacy of radiofrequency ablation in hepatocellular carcinoma patients. Hepatol Res. 2009;39(10):954–62. doi: 10.1111/j.1872-034X.2009.00537.x WOS:000270311300003. [DOI] [PubMed] [Google Scholar]
  • 32.Xie X, Wang X, Chen G. Clinical Effect of Ultrasound-guided Percutaneous Microwave Ablation in Treatment of Primary Small Hepatocellular Carcinoma. Clinical Misdiagnosis & Mistherapy. 2017;30(08):100–4. (in Chinese). [Google Scholar]

Decision Letter 0

Carlos Alberto Antunes Viegas

7 Jun 2024

PONE-D-24-15770Study on Microwave Ablation Temperature Prediction Model Based on Grayscale Ultrasound Texture and Machine LearningPLOS ONE

Dear Dr. Huang,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Jul 22 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Carlos Alberto Antunes Viegas, DVM; MSc; PhD

Academic Editor

PLOS ONE

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Thank you for stating the following in the Acknowledgments Section of your manuscript:

“This work was supported partly by National Natural Science Foundation of China (62271480), Youth Innovation Promotion Association CAS (2021324), Jiangsu Key Technology Research Development Program (BE2021612), Science and Technology Development Project of Suzhou (SYG202321), Postgraduate Research and Practice Innovation Program of Jiangsu Province (KYCX23_2093).”

We note that you have provided funding information that is currently declared in your Funding Statement. However, funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form.

Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows:

“This work was supported partly by National Natural Science Foundation of China (62271480), Youth Innovation Promotion Association CAS (2021324), Jiangsu Key Technology Research Development Program (BE2021612), Science and Technology Development Project of Suzhou (SYG202321), Postgraduate Research and Practice Innovation Program of Jiangsu Province (KYCX23_2093). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.”

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

3. We note that you have indicated that there are restrictions to data sharing for this study. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For more information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

Before we proceed with your manuscript, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information, data are owned by a third-party organization, etc.) and who has imposed them (e.g., a Research Ethics Committee or Institutional Review Board, etc.). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. For a list of recommended repositories, please see

https://journals.plos.org/plosone/s/recommended-repositories. You also have the option of uploading the data as Supporting Information files, but we would recommend depositing data directly to a data repository if possible.

We will update your Data Availability statement on your behalf to reflect the information you provide.

4. PLOS requires an ORCID iD for the corresponding author in Editorial Manager on papers submitted after December 6th, 2016. Please ensure that you have an ORCID iD and that it is validated in Editorial Manager. To do this, go to ‘Update my Information’ (in the upper left-hand corner of the main menu), and click on the Fetch/Validate link next to the ORCID field. This will take you to the ORCID site and allow you to create a new iD or authenticate a pre-existing iD in Editorial Manager. Please see the following video for instructions on linking an ORCID iD to your Editorial Manager account: https://www.youtube.com/watch?v=_xcclfuvtxQ

5. We are unable to open your Supporting Information file [S3 Fig.rar,S4 Fig.rar and S5 Fig.rar]. Please kindly revise as necessary and re-upload.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: No

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: No

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: 1.Is the selection of temperature measurement points related to the subsequent machine learning model? Why not select the vertical cross-section of the antenna as the temperature measurement point to predict the ablation radius? What is the necessity of vertically arranging 7 points? What is the significance of different distances between temperature measurement points in Figure 1?

2.On line 264, does Sv represent the cavitation area? It's best to have a unified statement

3.On line 278, x represents whether it represents Sv or Sp respectively. It is suggested to clarify the meaning of x

4.Comparing Figure 6 with Figure 7, it should be that the temperature is low, but there is an abnormal phenomenon at temperature measurement points 2 # and 3 #. How to explain this?

5.In the analysis of the results on line 305, did we conclude that the larger the prediction range, the greater the error? Or is it just a case study?

6.Figures 10 and 11 show Scatter plots of representative features with temperature at different power levels, with different conclusions. Therefore, whether there is a certain pattern for the important features commonly used in clinical microwave power levels of 60-100W or different heating times, or whether all the processes in the article need to be repeated, please provide the universality and significance of this study in the main text

7.There is too little introduction to the content of machine learning, which makes it difficult to fully understand the author's model, especially regarding how the first three features of importance ranking in section 4.4 were obtained?

Reviewer #2: General Comments:

MRI and CT treatment monitoring currently offer the greatest potential for thermal therapies to emerge into clinical practice. That said, given the associated high cost and limited accessibility, there is a need for cost-effective treatment monitoring tools like what is presented in this manuscript. Overall the temperature prediction method and results are interesting, but the image analysis results are somewhat confusing. A number of deficiencies need to be addressed.

First of all, it is not clear what treatment outcomes are being measured and compared in Fig 8. Throughout the manuscript the authors refer to ablation zone, vaporization zone, coagulation zone and cavitation area. The authors need to clearly state the tissue characteristics for these regions (what is going on) and if these regions of the thermal lesion are distinct or overlapping. It is clear that at the end of the heating (Fig. 8d,h) there is a region of tissue charring at the ablation needle surrounded by tissue coagulation. Without some description of the mechanisms for thermal lesion formation, it is difficult to interpret the data and analysis. For example, it is not clear in Tables 1 and 2 which data sets are to be compared. The microbubble zone is consistently larger than the tissue coagulation zone, but is this expected? In addition, the ML method consistently underestimates the coagulation area compared to the ImageJ data (used as a gold standard), but no explanation is provided. This needs to be discussed.

I'd also recommend to not refer to the formation of microbubbles as cavitation, as the mechanism here is MW heating, and is different from US induced cavitation. I found this confusing.

The images Fig 8d,h, clearly show tissue charring (temperatures in excess of 100 C, leading to water vaporization - microbubble formation). This is typically to be avoided during thermal therapy, due to the associated smoke production and the resulting change in tissue permittivity which alters antennae performance and power delivery. So a better test would be using MW powers that result in a maximum tissue temperature of 90 C.

In the Discussion (p.10), the author's state "The results indicate that this method can more effectively assess the ablation area.", but offer no justification, i.e., compared to what other method(s).

Specific Comments/Questions:

1. p.4, It is not clear what is meant by "texture information changes", please add a brief explanation.

2. What is the US frequency? The authors refer to a sampling frequency of 15 MHz, but that is not US frequency.

3. p.6, What type of thermocouple is used, Type-K, Type-T?

4. p.6, What are the dimensions of the mold (e.g what volume of liver is used in each experiment).

5. Fig 1B, It appears that the thermocouples are all not in the same plane. Please clarify. I'd also suggest changing in the caption “theoretical” to “desired” and include text to describe the temperature probes (blue) and ablation needle (red).

6. Fig 3 caption needs to be expanded to include a description of the individual process steps.

7. Fig 4 caption needs to be expanded to describe the image, e.g. An US frame showing the ablation needle and……..

8. Fig 5 caption needs to be more descriptive.

9. Fig 8 image, “width” not “wideth” and the caption needs to be more descriptive.

10. p. 10, The radiometric texture features used in the analysis are only referenced. These need to be described at least briefly in the manuscript for the reader to understand their utility.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2024 Sep 25;19(9):e0308968. doi: 10.1371/journal.pone.0308968.r002

Author response to Decision Letter 0


21 Jul 2024

Dear Editors,

We appreciate the opportunity to revise our manuscript entitled "Fast Ultrasonic Ablation Monitoring: An Innovative Approach Using Ultrasound RF Signals and Singular Value Decomposition" for consideration at PLOS ONE. We have carefully considered the reviewers' comments and have made the following revisions to our manuscript:

1. Style and File Naming: We have revised our manuscript according to PLOS ONE's style requirements, including the naming of files. We have ensured that all documents adhere to the guidelines provided in the style templates. Should there be any discrepancies we may have overlooked, we kindly ask for your guidance.

2. Acknowledgments and Funding Statement: We sincerely apologize for the oversight regarding the placement of funding information in the Acknowledgments section. We have removed the funding details from this section in our revised manuscript. We request that the following statement be retained in the Funding Statement section of the online submission form: "This work was supported partly by National Natural Science Foundation of China (62271480), Youth Innovation Promotion Association CAS (2021324), Jiangsu Key Technology Research Development Program (BE2021612), Science and Technology Development Project of Suzhou (SYG202321), Postgraduate Research and Practice Innovation Program of Jiangsu Province (KYCX23_2093). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript."

3. Data Sharing: After thorough discussion, we have decided to make our data publicly available. The primary data related to our study have been uploaded as Supporting Information. Given the large volume of our original experimental data, which may not be suitable for deposition in a public database, we have provided access to this data through our submission. We welcome any inquiries for further experimental details via email correspondence.

4. ORCID iD: We acknowledge the requirement for the corresponding author to have an ORCID iD linked to the Editorial Manager account. We will ensure that this is done promptly.

5. Supporting Information File: We have converted the image groups from the compressed files [S3 Fig.rar, S4 Fig.rar, and S5 Fig.rar] into PDF format for ease of access and have re-uploaded them. We believe this will resolve the issue of file accessibility.

We trust that these revisions meet the journal's requirements and enhance the quality of our submission. We look forward to your positive response.

Sincerely,

Professor Yan Huang

Department of Ultrasound, Nanjing Hospital of Chinese Medicine Affiliated to Nanjing University of Chinese Medicine, Nanjing, China

E-mail: jacob6666@163.com

July 21, 2024

Dear Editor:

Thank you very much for the valuable comments on our manuscript, “Fast Ultrasonic Ablation Monitoring: An Innovative Approach Using Ultrasound RF Signals and Singular Value Decomposition”. We have made every effort to revise the manuscript carefully, in light of the reports from the reviewers and editors.

We sincerely hope the revised manuscript could be considered for publication on PLOS ONE. Really appreciate your time and kind consideration. Detailed responses can be found as the follows.

Best Regards!

Yan Huang

Reviewer #1:

1. Is the selection of temperature measurement points related to the subsequent machine learning model? Why not select the vertical cross-section of the antenna as the temperature measurement point to predict the ablation radius? What is the necessity of vertically arranging 7 points? What is the significance of different distances between temperature measurement points in Figure 1?

Response: Thank you for your question. Our selection of temperature measurement points is closely related to the machine learning model, aiming to enhance the model's ability to learn temperature variations within the ablation zone using data from different locations. Simulation images reveal the non-uniformity of temperature distribution during ablation, confirming the necessity of setting measurement points at various positions.

We chose the cross-section parallel to the ablation needle for temperature measurement points because this cross-section is commonly used in clinical practice and provides richer tissue information. Additionally, as long as the long axis of the ablation needle is centered in the ultrasound image, changing the probe direction will not affect the accuracy of the ablation radius measurement. In contrast, the vertical cross-section can lead to measurement deviations when the probe position changes. Simulation images show that the ablation area forms an ellipsoid in three-dimensional space, and the ultrasound probe can only scan one section of it, leading to the aforementioned issues.

Simulation 2D/3D diagram

ZHU Roujun, et al. "Design of Precision Treatment Planning System for Tumor Microwave Thermal Ablation Based on LabWindows,"(in Cineses), Life Science Instruments, vol. 19, no. 01, 2021, pp. 68-74.

Arranging the temperature measurement points vertically ensures they are in the same plane as the ablation needle in the ultrasound image. This helps us accurately determine the position of the temperature probes and ensures that the data collected from each measurement point corresponds to the correct ultrasound ROI.

The distances between the temperature measurement points are carefully designed to meet practical needs. To collect more temperature data and improve the model's accuracy, we aim to place as many measurement points as possible within a limited space. However, spacing them too closely would cause acoustic shadow interference between the needles, affecting grayscale signal collection. Therefore, the uniform spacing of the temperature measurement points in Figure 1B ensures an optimal number of measurement points while maintaining image quality.

2. On line 264, does Sv represent the cavitation area? It's best to have a unified statement.

Response: Sv represents the hyperechoic "cavitation area" observed on the ultrasound image. However, we have noticed that the article contains multiple concepts related to ablation and cavitation, which may confuse readers. To address this issue, we have conducted a thorough review and revision of the entire text, clearly distinguishing between the tissue effects caused by microwave heating and the cavitation phenomenon induced by ultrasound. We have unified the terminology to avoid ambiguity:

�The terms "ablation zone," "ablation area," and "ablation range" have been consistently revised to "coagulation zone" or "coagulation area" to clearly indicate the extent of tissue coagulation following microwave ablation therapy.

�The terms "vaporization zone" and "cavitation area" have been unified as "hyperechoic area" to describe the high-echo phenomenon observed in ultrasound images due to the formation of gas within the tissue.

3. On line 278, x represents whether it represents Sv or Sp respectively. It is suggested to clarify the meaning of x

Response: Thank you for your suggestion. We have clarified the meaning of x in the revised manuscript (Lines 335-336).

4. Comparing Figure 6 with Figure 7, it should be that the temperature is low, but there is an abnormal phenomenon at temperature measurement points 2 # and 3 #. How to explain this?

Response: The temperature measurement points selected in the 15 W power group represented by Figure 6 have a broader prediction range, and theoretically, their overall temperature should be lower than that of the 20 W power group. However, due to the different distances of the seven measurement points from the high-temperature center, the points closer to the high-temperature center will naturally record higher temperatures. Consequently, some measurement points in the 15 W power group, such as 2# and 3#, actually recorded higher temperatures than certain measurement points in the 20W power group.

5. In the analysis of the results on line 305, did we conclude that the larger the prediction range, the greater the error? Or is it just a case study?

Response: We did observe a trend where temperature measurement points with larger prediction ranges exhibited greater prediction errors. This conclusion is not merely a result of a case study but is derived from our comprehensive analysis of the entire dataset. During data processing, we noticed that when the temperature range at the measurement points increased, particularly when more high-temperature segments were included, the model's prediction error tended to increase. This phenomenon was observed in data from multiple measurement points and was not limited to isolated cases. We believe this is because, in the high-temperature range, the changes in tissue grayscale signals can be more complex, making accurate predictions more challenging for the model. Therefore, we conclude that as the temperature prediction range expands, especially when it includes more high-temperature data, the prediction error tends to increase, as reflected in the higher mean absolute error.

6. Figures 10 and 11 show Scatter plots of representative features with temperature at different power levels, with different conclusions. Therefore, whether there is a certain pattern for the important features commonly used in clinical microwave power levels of 60-100W or different heating times, or whether all the processes in the article need to be repeated, please provide the universality and significance of this study in the main text.

Response: We recognize that different texture features may emerge at various clinical microwave power levels (60-100W) and with varying heating times. We agree that these features are influenced not only by power but also by tissue characteristics. Although there are differences between porcine liver and human liver tissue properties, which might yield different results, we believe the biological significance behind texture features remains consistent.

In our study, cluster prominence, variance, mean absolute deviation, and minimum were identified as highly important features in both ablation groups, indicating their key roles in temperature prediction. The correlation analysis of these features with temperature revealed the following patterns: the minimum value was positively correlated with temperature, while the other features showed a negative correlation. Specifically, an increase in the minimum value reflected a general rise in grayscale values within the ROI as temperature increased. In contrast, cluster prominence, variance, and mean absolute deviation, which measure skewness and asymmetry, decreased with rising temperatures, indicating that tissue coagulation and bubble formation made the grayscale images more homogeneous.

We understand that due to variations in experimental conditions and tissue characteristics, the features with high-temperature correlation might differ across studies. However, we believe one core conclusion is universal: as temperature increases, the grayscale values of ablated tissue increase, and the images become more homogeneous. For instance, in the literature cited in our discussion, Wang et al. observed that in ex vivo porcine liver heated in a water bath, texture feature parameters such as the mean grayscale of GLH, homogeneity of GLCM, hybrid entropy, inverse difference moment, and correlation of GGCM were closely related to temperature and increased with it. These changes in texture features suggest that similar patterns can be observed across different studies, supporting our conclusion.

To further investigate the universality of these features under different conditions, we plan to extend the power range and heating times in future studies, as well as include in vivo experiments to verify whether the above conclusions hold. We believe that through these additional studies, we can better understand the applicability of these features in various clinical scenarios.

We appreciate your suggestion and have incorporated the discussion on the universality and significance of the study into the revised manuscript in lines 491-509.

7. There is too little introduction to the content of machine learning, which makes it difficult to fully understand the author's model, especially regarding how the first three features of importance ranking in section 4.4 were obtained?

Response: In this study, feature importance was assessed using the Gini importance index from the random forest algorithm. The Gini importance index measures a feature's ability to reduce impurity at each split point within the random forest. A high Gini coefficient indicates that the feature provides more information during classification or regression, thus contributing more significantly to the model's predictive power. By calculating the Gini coefficient for each feature, we ranked their importance accordingly. In Section 4.4, we utilized machine learning methods to analyze the top three features based on their Gini coefficients. These features, identified as having the highest importance rankings, were subjected to detailed analysis. The rankings were consistent with those in Figure 5, which displays the importance of features for different power groups. These top features were identified due to their strong correlation with temperature changes, making them significant predictors in our model.

To provide a more comprehensive understanding of our model, we have supplemented the revised manuscript with additional details on the feature importance evaluation method (Lines 265-271).

Reviewer #2: MRI and CT treatment monitoring currently offer the greatest potential for thermal therapies to emerge into clinical practice. That said, given the associated high cost and limited accessibility, there is a need for cost-effective treatment monitoring tools like what is presented in this manuscript. Overall the temperature prediction method and results are interesting, but the image analysis results are somewhat confusing. A number of deficiencies need to be addressed.

General comments:

1. First of all, it is not clear what treatment outcomes are being measured and compared in Fig 8. Throughout the manuscript the authors refer to ablation zone, vaporization zone, coagulation zone and cavitation area. The authors need to clearly state the tissue characteristics for these regions (what is going on) and if these regions of the thermal lesion are distinct or overlapping. It is clear that at the end of the heating (Fig. 8d,h) there is a region of tissue charring at the ablation needle surrounded by tissue coagulation. Without some description of the mechanisms for thermal lesion formation, it is difficult to interpret the data and analysis. For example, it is not clear in Tables 1 and 2 which data sets are to be compared. The microbubble zone is consistently larger than the tissue coagulation zone, but is this expected? In addition, the ML method consistently underestimates the coagulation area compared to the ImageJ data (used as a gold standard), but no explanation is provided. This needs to be discussed.

Response: Thank you for your question. To enhance the clarity of our manuscript and eliminate conceptual confusion, we have thoroughly revised and clarified the key terms mentioned in the text. We have standardized the use of "coagulation zone" to refer to the tissue area where coagulation necrosis has occurred following microwave ablation. Additionally, we have unified "vaporization zone" and "cavitation area" under the term "hyperechoic area," which describes the region in the ultrasound grayscale image exhibiting high echogenicity due to gas formation within the tissue.

In our study:

�The coagulation zone represents the area of protein denaturation and necrosis, which includes the charred region close to the ablation needle where temperatures are highest.

�The hyperechoic area, observed in ultrasound images, is typically larger than the actual coagulation zone because it covers the region affected by gas bubbles formed during ablation.

To address the question regarding the comparison of data sets in Tables 1 and 2, we have included additional explanations in the revised manuscript (Lines 407-420). Tables 1 and 2 compare the coagulation zone areas

Decision Letter 1

Carlos Alberto Antunes Viegas

5 Aug 2024

Study on Microwave Ablation Temperature Prediction Model Based on Grayscale Ultrasound Texture and Machine Learning

PONE-D-24-15770R1

Dear Dr. Yan Huang,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Carlos Alberto Antunes Viegas, DVM; MSc; PhD

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Dear authors

Your article is ready to be published, but you must consider the first reviewer's suggestion regarding the second round of revisions in the last version.

Best Regards

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: (No Response)

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Comparing Figure 6 with Figure 7 and Fig.1 , I think the positions of thermocouples and microwave antennas are fixed in every experiments, so there is no ” However, due to the different distances of the seven measurement points from the high-temperature center“ as author explained. On the contrary, if the positions of the two are not fixed, is the temperature measurement point data in machine learning reliable?

Reviewer #2: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

**********

Acceptance letter

Carlos Alberto Antunes Viegas

16 Sep 2024

PONE-D-24-15770R1

PLOS ONE

Dear Dr. Huang,

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now being handed over to our production team.

At this stage, our production department will prepare your paper for publication. This includes ensuring the following:

* All references, tables, and figures are properly cited

* All relevant supporting information is included in the manuscript submission,

* There are no issues that prevent the paper from being properly typeset

If revisions are needed, the production department will contact you directly to resolve them. If no revisions are needed, you will receive an email when the publication date has been set. At this time, we do not offer pre-publication proofs to authors during production of the accepted work. Please keep in mind that we are working through a large volume of accepted articles, so please give us a few weeks to review your paper and let you know the next and final steps.

Lastly, if your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

If we can help with anything else, please email us at customercare@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Carlos Alberto Antunes Viegas

Academic Editor

PLOS ONE

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Supplementary Materials

    S1 Table. Predicted and measured temperatures for the 15 W power group.

    (XLSX)

    pone.0308968.s001.xlsx (41.2KB, xlsx)
    S2 Table. Predicted and measured temperatures for the 20 W power group.

    (XLSX)

    pone.0308968.s002.xlsx (40.4KB, xlsx)
    S1 Fig. Cross-sectional photos of liver post-ablation for 15 W and 20 W power groups.

    (PDF)

    pone.0308968.s003.pdf (962KB, pdf)
    S2 Fig. Thermal maps for 15 W and 20 W power groups: Machine learning predictions and 54°C threshold highlights.

    (PDF)

    pone.0308968.s004.pdf (686.9KB, pdf)
    S3 Fig. Ultrasound greyscale images at the end of ablation for 15 W and 20 W power groups.

    (PDF)

    pone.0308968.s005.pdf (674.7KB, pdf)

    Data Availability Statement

    All relevant data are in the paper and its Supporting Information files.


    Articles from PLOS ONE are provided here courtesy of PLOS

    RESOURCES