Skip to main content
PLOS One logoLink to PLOS One
. 2021 Mar 4;16(3):e0247330. doi: 10.1371/journal.pone.0247330

Prediction of femoral osteoporosis using machine-learning analysis with radiomics features and abdomen-pelvic CT: A retrospective single center preliminary study

Hyun Kyung Lim 1, Hong Il Ha 2,*, Sun-Young Park 2, Junhee Han 3
Editor: Alfredo Vellido4
PMCID: PMC7932154  PMID: 33661911

Abstract

Background

Osteoporosis has increased and developed into a serious public health concern worldwide. Despite the high prevalence, osteoporosis is silent before major fragility fracture and the osteoporosis screening rate is low. Abdomen-pelvic CT (APCT) is one of the most widely conducted medical tests. Artificial intelligence and radiomics analysis have recently been spotlighted. This is the first study to evaluate the prediction performance of femoral osteoporosis using machine-learning analysis with radiomics features and APCT.

Materials and methods

500 patients (M: F = 70:430; mean age, 66.5 ± 11.8yrs; range, 50–96 years) underwent both dual-energy X-ray absorptiometry and APCT within 1 month. The volume of interest of the left proximal femur was extracted and 41 radiomics features were calculated using 3D volume of interest analysis. Top 10 importance radiomic features were selected by the intraclass correlation coefficient and random forest feature selection. Study cohort was randomly divided into 70% of the samples as the training cohort and the remaining 30% of the sample as the validation cohort. Prediction performance of machine-learning analysis was calculated using diagnostic test and comparison of area under the curve (AUC) of receiver operating characteristic curve analysis was performed between training and validation cohorts.

Results

The osteoporosis prevalence of this study cohort was 20.8%. The prediction performance of the machine-learning analysis to diagnose osteoporosis in the training and validation cohorts were as follows; accuracy, 92.9% vs. 92.7%; sensitivity, 86.6% vs. 80.0%; specificity, 94.5% vs. 95.8%; positive predictive value, 78.4% vs. 82.8%; and negative predictive value, 96.7% vs. 95.0%. The AUC to predict osteoporosis in the training and validation cohorts were 95.9% [95% confidence interval (CI), 93.7%-98.1%] and 96.0% [95% CI, 93.2%-98.8%], respectively, without significant differences (P = 0.962).

Conclusion

Prediction performance of femoral osteoporosis using machine-learning analysis with radiomics features and APCT showed high validity with more than 93% accuracy, specificity, and negative predictive value.

Introduction

As the elderly population has rapidly grown, osteoporosis has increased and developed into a serious public health concern [1]. Approximately 30% of all postmenopausal women have osteoporosis in the developed countries, and up to 50% of these patients will sustain one or more osteoporotic fracture in their life time [2]. Although the prevalence of osteoporosis is very high, it has specific diagnosis tool such as dual-energy X-ray absorptiometry (DXA), effective treatment options, and preventive methods [1]. Therefore, osteoporosis is a disease in which screening can have a great effect on patient outcomes [3]. However, screening for osteoporosis using DXA has been underperformed because osteoporosis is asymptomatic until major incidental fragile fractures occur, such as vertebral body or hip fractures [2]. Patients often do not recognize the seriousness of this disease and, therefore, do not participate in the screening program voluntarily [4]. There is a growing consensus regarding the need for alternative screening methods to overcome the limitations and underuse of DXA as a screening method for osteoporosis. Abdomen-pelvic computed tomography (APCT) is commonly performed on adults to evaluate various diseases, during routine health check-ups or follow-up diagnosed diseases. Even if a small number of these scans were used to opportunistically screen for osteoporosis, the impact could be substantial. Several studies have shown optimistic results using APCT for opportunistic screening for osteoporosis [57].

Radiomics is the most advanced application within the radiology research field. It extracts various features from medical images and has the potential to find disease characteristics that fail to be appreciated by the naked eye using specially designed data-characterization algorithms for image analysis [8]. These radiomics features are the distinctive imaging features between disease forms might be useful for predicting prognosis and therapeutic response for various conditions, thus providing valuable information for personalized therapy [9, 10]. As osteoporosis progresses, bone mineral density (BMD) is decreased and bony microstructure change occur, simultaneously [11]. BMD-decrease is highly correlated with mean computed tomography Hounsfield unit (CTHU) change, as well as HU histogram analysis (HUHA) of the proximal femur volume representing fatty marrow content (HUHAFat, percentage ratio of HU range ≤ 0HU) or thick cortical bone content (HUHABone, percentage ratio of HU range ≥126HU) [5, 12]. This radiomics analysis may be useful for the evaluation of microstructure changes of trabecular bone [1315]. Additionally, machine-learning analysis is changing the paradigm of medical practice, which is best suited for mass screening [16]. However, to the best of our knowledge, there has been no research evaluating femoral osteoporosis using radiomics features and machine learning analysis. Therefore, the purpose of this study was to evaluate the predicting performance of machine learning analysis for diagnosing femoral osteoporosis using radiomics features and APCT.

Materials and methods

This retrospective study was approved by institutional review board and ethics committee at Hallym University Sacred Heart hospital and the need for informed consent was waived.

Patients

Between July 2018 and June 2019, 569 patients aged 50 years or older who had undergone APCT and DXA within an interval of a 1-month period (mean, 3.8 ± 6.1 days; range, 0–30 days) were retrospectively included. Among these patients, 69 were excluded due to bone metastases (n = 8), metastasis other than bone (n = 10), history of receiving chemotherapy within the last 3 months (n = 26), primary bone disease (e.g., fibrous dysplasia; n = 4), developmental or traumatic deformation of the femur (n = 6), or any total hip arthroplasty or internal nailing (n = 15). Finally, 500 patients (mean age, 66.5 ± 11.8 yrs; range, 50–96 years) were included. This cohort consisted of 70 men (mean age, 72.6 ± 8.0 yrs; range, 54–89 years) and 430 women (mean age, 65.4 ± 12.1 yrs; range, 50–96 years). There were no duplicate patients enrolled. The reasons for CT imaging were as follows: cancer metastasis surveillance (n = 327), minor trauma (e.g., slip-down injury or simple fall-down injury; n = 37), or routine health check-up or medical inspection (n = 136). Among the included patients, 70% were randomly selected to be the training cohort (M:F = 48:302; age, 66.8±12.2 yrs), and the remaining 30% were selected to be the validation cohort (M:F = 22:128; age, 65.7±11.1 yrs) (Fig 1).

Fig 1. Flowchart of patient enrollment and random selection for machine-learning analysis.

Fig 1

Dual-energy X-ray absorptiometry

DXA of the proximal femur for BMD assessment was performed using a single BMD scanner (GE Healthcare Lunar Prodigy Densitometers, Madison, WI, USA). The lowest T-score of the femoral neck was used as the reference standard. Osteoporosis of the femur was defined as a T-score ≤ −2.5, and non-osteoporosis was defined as a T-score > −2.5 (1).

Computed tomography imaging

All CT examinations were performed using two multidetector-row CT scanners (SOMATOM Definition Edge, SOMATOM Definition Flash; Siemens Healthcare, Forchheim, Germany) in the standard single-energy CT mode. Automatic tube voltage selection (Care kVp) and automatic tube current modulation (CARE Dose 4D) protocols were applied. To exclude the effect of the contrast agent on the CTHU measurement, all measurements were performed on only pre-contrast CT scans [17]. The scanning parameters were as follows: detector collimations, 128 × 0.6 mm; pitch, 0.6; gantry rotation time, 0.5 s; tube current, 200 or 289 mAs; tube voltage, 100 or 120 kVp; and iterative reconstruction (sinogram-affirmed iterative reconstruction, S1, I40f). The voxel size of all raw data was 0.67mm x 0.67mm x 1mm.

Radiomics analysis

The radiomics analysis was performed using two commercial software programs (Aquarius iNtuition v4.4.12®, TeraRecon, Foster City, CA, USA; Medip®, Medical Imaging Solution for Segmentation and Texture Analysis, Korea). Each target volume of interest (VOI) of the left femur was extracted using the region growing editing tool. The area below the lesser trochanter of the femur was excluded to maintain a constant VOI across all measurement (Fig 2). For each VOI, 41 radiomic features were extracted and divided into four groups: (1) first-order grey-level histogram features to describe the distribution of grey-values within the volume; (2) geometric features to describe the shape and size of the volume of interest; (3) grey-level co-occurrence level matrices are statistical features used to explore the spatial relationship between two pixels with certain distance and direction, including contrast relation, entropy, angular second moment, etc.; and (4) wavelet transformation is a transformation that separates data into different frequency components, and then examines each component with resolution matched to its scale. A detailed description of these radiomics features are independent of imaging modality and can be found in the literature [1820].

Fig 2. Measurement of the 3D volume of proximal femur.

Fig 2

(A) Texture analysis. Forty-one features are extracted from the volume of interest (red). (B) Mean CTHU and HU histogram analysis (HUHA) are simultaneously calculated from the other commercial 3D image processing software. Negative HU range (yellow box, HUHAFat, ≤0HU) is considered as fatty marrow content and equal or more than 126 HU range (red box, HUHABone) is considered as dense bone content, respectively. (A) Texture analysis. (B) Mean-CTHU and HU histogram analysis (HUHA).

Feature extraction and random forest model

Our study design followed the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) guidelines [21]. To overcome the repeatability weakness of radiomics, only those with an intraclass correlation coefficient (ICC) value higher than 0.9 were considered stable and selected for subsequent analysis. We used the random forest (RF) algorithm of various machine-learning analysis methods. Although the RF algorithm itself enables the efficient selection of the highly correlated variables and reduces the number of variables, further feature selection was performed by the Mean Decrease in Gini index, and the top 10 important features were selected (Fig 3) [22]. The RF algorithm is an ensemble of unpruned classification or regression trees created by using bootstrap samples of the training data and random feature selection in tree induction. Predictions are made by aggregating the predictions of the ensemble. We randomly selected 70% (n = 350) of the samples as the training cohort and the remaining 30% (n = 150) as the validation cohort using a ‘caret’ R package [23]. An RF is a meta-estimator that fits a number of decision-tree classifiers on various sub-samples of the dataset. After each tree, the decision for classification result was determined, and the result was averaged to improve the predictive accuracy and control over-fitting. In the RF algorithm, each tree in the ensemble is built from a sample drawn with a replacement from the training cohort. In addition, when splitting a node during the construction of the tree, the split that is picked is the best split among a random subset of the features. For the best hyperparameter tuning, 5-fold cross validation using random search was performed and the result was summarized in S1 Table. The hyperparameter of our RF algorithm were as follows: mtry = 3, minimum nodal size = 11, and splitrule = extratrees. The best model was selected and validated in the test cohort. Further explanation of RF model and features was summarized in S1 Appendix.

Fig 3. Top 10 important features with mean decrease in Gini index (A) and distribution of minimal depth (B).

Fig 3

A higher mean decrease in Gini index indicates higher features importance. The distribution of minimal depth is marked by a vertical bar with a value label on the trees of random forest. (A) Mean decrease in Gini index. (B) Distribution of minimal depth.

Statistical analysis

All statistical analyses were performed with the open-source statistical computing environment R (version 3.6.1; R Foundation for Statistical Computing) and the Medcalc Statistical Software version 19.1.3 (MedCalc Software bv, Ostend, Belgium). The reproducibility of the radiomics features was evaluated by ICC using a two-way random model with absolute measurements. To assess ICC, two radiologists (one with 12 years of experience interpreting body images and the other with 6 years of experience interpreting musculoskeletal images) measured the radiomics features in images from 40 randomly selected cases. Using the diagnostic test confusion matrix, the prediction accuracy of the training and validation cohorts were calculated. The area under curve (AUC) of the receiver operating characteristic curve and 95% confidence interval (CI) of the training and validation cohorts were calculated. The AUCs of training and validation models were compared using the method developed by DeLong et al. A P-value < 0.05 was considered a significant difference.

Results

The demographic information of the study cohort is summarized in Table 1. Overall, 104 patients were diagnosed with osteoporosis, and the osteoporosis prevalence of this cohort was 20.8%.

Table 1. Comparison of demographics between the osteoporosis and non-osteoporosis groups.

Osteoporosis (n = 104) Non-osteoporosis (n = 396) P-value
Sex (M:F) 8:96 62:334 < 0.001
Age (years, mean ± SD) 78.6 ± 8.6 63.3 ± 10.5 < 0.000
T-score -3.11 ± 0.53 -0.80 ± 1.01 < 0.000
BMD (g/cm2) 0.57 ± 0.06 0.84 ± 0.12 < 0.000
BMI (kg/m2) 22.1 ± 3.7 24.4 ± 4.1 < 0.000
Interval between DXA to APCT (day) 6.5 ± 7.2 3.1 ± 5.5 < 0.000

APCT = abdominal-pelvic CT; BMD = bone material density; BMI, body mass index; DXA = dual-energy X-ray absorptiometry.

The AUC and correlation coefficient to predict femoral osteoporosis and the ICC of all radiomics features are summarized in Table 2. According to the RF feature selection algorithm made by the mean decrease in Gini index and distribution of minimal depth, waveletLLL, HUHABone, mean-CTHU, HUHAFat, waveletHLL, kurtosis, waveletLLH, waveletLHL, texture energy, moment, and skewness were selected as the top important radiomics features (Fig 3).

Table 2. Summary of AUC and correlation coefficient to predict femoral osteoporosis, and intraclass correlation coefficient of all radiomics features.

Radiomic Feature AUC (95% CI) Correlation coefficient ICC (95% CI)
First-order gray-level histogram (n = 9)
Entropy 0.884 (0.853–0.911) -0.573 0.993 (0.987–0.996)
HUHA_bone 0.950 (0.927–0.967) -0.327 0.999 (0.999–0.999)
HUHA_fat 0.938 (0.908–0.953) 0.699 0.997 (0.995–0.998)
Kurtosis 0.903 (0.873–0.927) 0.621 0.993 (0.988–0.996)
Mean_CTHU 0.951 (0.928–0.968) -0.628 0.989 (0.903–0.996)
Skewness 0.876 (0.844–0.903) 0.544 0.991 (0.984–0.994)
Texture_energy 0.877 (0.845–0.904) -0.470 0.987 (0.973–0.992)
Uniformity 0.866 (0.833–0.895) 0.549 0.994 (0.990–0.996)
Variance 0.852 (0.817–0.882) -0.505 0.981 (0.966–0.989)
Geomatric features (n = 6)
Discrete Compactness 0.562 (0.517–0.606) 0.075 0.979 (0.962–0.987)
Effective diameter 0.501 (0.456 to 0.545) -0.010 0.934 (0.882–0.960)
Roundness 0.723 (0.682–0.762) -0.325 0.953 (0.917–0.973)
Sphericity 0.570 (0.523–0.612) -0.140 0.977 (0.958–0.986)
Texture_compactness1 0.582 (0.538–0.626) -0.113 0.881 (0.795–0.931)*
Texture_compactness2 0.701 (0.659–0.741) -0.292 0.843 (0.738–0.908)*
Co-occurrence matrix (n = 18)
CROSS_GLCMASM 0.776 (0.737–0.812) 0.466 0.993 (0.986–0.995)
CROSS_GLCMcontrast 0.683 (0.661–0.743) -0.1226 0.969 (0.942–0.983)
CROSS_GLCMentropy 0.821 (0.766–0.837) -0.452 0.987 (0.974–0.993)
CROSS_GLCMIDM 0.662 (0.619–0.703) 0.229 0.998 (0.995–0.998)
EW_GLCMASM 0.782 (0.744–0.818) 0.467 0.992 (0.985–0.995)
EW_GLCMcontrast 0.667 (0.645–0.728) -0.119 0.960 (0.926–0.977)
EW_GLCMentropy 0.775 (0.754–0.827) -0.438 0.987 (0.974–0.993)
EW_GLCMIDM 0.682 (0.639–0.723) 0.270 0.998 (0.995–0.998)
Homogeneity 0.787 (0.748 to 0.822) 0.409 0.996 (0.993–0.998)
Moment 0.757 (0.716–0.794) 0.446 0.980 (0.965–0.988)
NS_GLCMASM 0.760 (0.721–0.797) 0.446 0.992 (0.985–0.995)
NS_GLCMcontrast 0.692 (0.668–0.749) -0.122 0.978 (0.956–0.988)
NS_GLCMentropy 0.757 (0.734–0.809) -0.408 0.983 (0.966–0.991)
NS_GLCMIDM 0.640 (0.596–0.682) 0.182 0.998 (0.995–0.998)
SIX_GLCMASM 0.803 (0.765–0.837) 0.495 0.993 (0.987–0.996)
SIX_GLCMcontrast 0.769 (0.729–0.805) -0.244 0.966 (0.935–0.981)
SIX_GLCMentropy 0.835 (0.800–0.867) -0.500 0.988 (0.974–0.993)
SIX_GLCMIDM 0.686 (0.643–0.726) 0.294 0.998 (0.995–0.998)
Wavelet transformation (n = 8)
Wavelet_HHH 0.626 (0.582–0.668) -0.175 0.379 (0.109–0.596)*
Wavelet_HHL 0.795 (0.757–0.830) -0.397 0.642 (0.444–0.780)*
Wavelet_HLH 0.767 (0.728–0.803) -0.366 0.618 (0.410–0.765)*
Wavelet_HLL 0.927 (0.900–0.948) -0.583 0.966 (0.940–0.980)
Wavelet_LHH 0.731 (0.690–0.769) -0.334 0.607 (0.395–0.757)*
Wavelet_LHL 0.915 (0.887–0.938) -0.565 0.956 (0.923–0.975)
Wavelet_LLH 0.912 (0.884–0.935) -0.550 0.949 (0.910–0.970)
Wavelet_LLL 0.950 (0.928–0.968) -0.640 0.999 (0.998–0.999)

Values in parentheses mean 95% confidential interval.

* Values are excluded in random forest analysis.

The prediction accuracy of the machine-learning analysis using the RF model to diagnose osteoporosis in training and validation cohorts are summarized in Table 3. Both cohorts showed more than 80% of sensitivity, 94% of specificity, 94% of negative predictive value (NPV), and 93% of accuracy. Training model showed 95.9% of AUC (95% CI, 93.7%-98.1%) and validation model showed 96.0% of AUC (95% CI, 93.2%-98.8%). There was no significant difference of AUCs predicting femoral osteoporosis between training and validation models (P = 0.962) (Fig 4).

Table 3. Diagnostic performance of the Machine-learning analysis.

Training cohort (n = 350) Validation cohort (n = 150)
Sensitivity (%) 86.6 (76.1–93.7) 80.0 (61.4–92.3)
Specificity (%) 94.5 (91.0–96.7) 95.8 (90.5–98.6)
Positive likelihood ratio 15.3 (9.4–24.9) 19.2 (8.0–46.1)
Negative likelihood ratio 0.1 (0.1–0.3) 0.2 (0.1–0.43)
Disease prevalence (%) 19.1 20.0
Positive predictive value (%) 78.4 (69.1–85.5) 82.8 (66.6–92.0)
Negative predictive value (%) 96.7 (94.2–98.1) 95 (90.4–97.5)
Accuracy (%) 92.9 (89.6–95.3) 92.7 (87.3–96.3)

ǂValues in parentheses mean 95% confidential interval.

Fig 4. Comparison of area under curves of training and validation cohorts.

Fig 4

Discussion

The primary goal of this study was to evaluate the prediction accuracy of osteoporosis using machine-learning analysis with radiomics features and APCT. In this study, the prediction accuracy of osteoporosis was 95.9% and 96.0% in the training and validation cohorts, respectively. The predicting performance for diagnosis of osteoporosis was 95.8% in specificity, 95% in NPV, 80% in sensitivity, and 92.7% in diagnostic accuracy at validation cohort. In particular, our results showed high specificity and NPV more than 95%, which is considered as meaningful results to select healthy peoples. Therefore, screening using this method may contribute to reduce or prevent the unnecessary duplication check and cost of DXA.

In order to increase the osteoporosis screening rate, several policies have been tried such as patient selection using questionnaires, education of primary clinic physicians, coverage by medical insurance, and so on [2429]. However, more than 25 million abdominal-pelvic CT scans have been performed on adults each year in the United States. If osteoporosis screening with APCT is possible, the effect would be enormous. Based on this concept, several studies have shown optimistic results. Most studies have reported the usefulness of osteoporosis diagnosis by mean CTHU measurement [57, 30]. In some studies, images were analyzed using HU histogram analysis and texture analysis [5, 31, 32]. Recently, a few studies on the usefulness of osteoporosis screening using artificial intelligence have been published [6, 33] [new ref DII]. The advantage of precision medicine using artificial intelligence is that auto-segmentation and mass-screening using big data is possible (10, 11). In this study, femur segmentation was performed by researchers using a semi-automatic extraction using 3D image analysis software. However, the auto-segmentation algorithm of the femur has been recently developed and applied to image analysis for research purpose. If actual clinical application is made, osteoporosis screening can be easier and more effective and may improve patient convenience. Although there may be concerns about radiation exposure, the concept of opportunistic screening using APCT is to obtain additional information related to osteoporosis from already-performed CT scans for other medical reasons. As a result, patients can simultaneously obtain secondary bone health information without additional radiation exposure, thereby gaining additional benefits in terms of cost, time, and convenience.

We used top 10 radiomics features based on the RF feature selection. Among them, mean-CTHU is the typical feature in conventional CT image analysis. This conventional feature was selected as a meaningful feature with high feature importance scores. HUHAFat and HUHABone features are based on the HU number distribution and represented specific tissue contents such as fatty marrow and dense bone content, respectively. Kurtosis and skewness measure the peakedness and symmetry of the HU histogram. The effective diameter was defined as the diameter of a sphere whose volume is equal to the segmented volume. The wavelet features are image transform technique based on the space–frequency decomposition with low computational complexity [18, 19]. In addition to the mean-CTHU as a conventional feature, these radiomic features were selected as important features evaluating osteoporosis and thought to reflect the pathophysiology of osteoporosis not detectable by naked human eyes.

We analyzed the left femur from the head to lesser trochanter as a VOI because this area is consistent with the DXA target range. As the proximal femur has a 3D complex structure, 3D image analysis would be useful for evaluating osteoporosis, rather than 2D image analysis, and would exclude the observer’s subjection to select the target image. Although the target VOI range was arbitrary set with semi-auto-segmentation of the 3D analysis software, our results proved this target VOI set was appropriate and reproducible. More precise target volume selection focusing the femoral neck or Ward’s triangle would likely improve the diagnostic accuracy; however, the current VOI selection proved effective to predict osteoporosis because this method was easy, simple, and highly reproducible.

In this study, among the forty-one radiomic features, only six radiomics features were excluded because of low reproducibility. Most features showed a high reproducibility of more than 0.9 of ICC. Advantage of the 3D image software is high reproducibility [34, 35]. As the femur shows a high contrast with the surrounding tissue, automatic segmentation is relatively easy. Therefore, it is expected that automatic osteoporosis prediction can be implanted on a picture archiving and communication system or workstation of CT consoles through automatic femur segmentation and machine-learning analysis during CT acquisition and post-image processing. Furthermore, we researched the prediction of osteoporosis by applying machine learning analysis using specific radiomics features and HUHA due to the limited number of patients in this study. Recent AI research is moving to deep learning, which is the evolution of machine learning and it helps in making better precision medicine than machine learning. Deep learning is similar to machine learning, but it does not require artificial intervention. However, it requires big data to train the model otherwise it won’t work as expected. If osteoporosis screening can be implemented by deep learning using wide and easily accessible plain radiographs or large number of CT images in the future, great progress can be expected in the prevention and treatment of osteoporosis.

Although this study achieved a high prediction accuracy, the major limitation was these results were obtained from a single center and single race. Osteoporosis differs according to gender and race [36]. However, our results were based on the DXA results within a 1-month interval, the only standard reference of osteoporosis diagnosis. Based on our research results, it is necessary to prove the validity through prospective multi-center and multi-ethnic studies. Osteoporosis is divided into three groups according to the DXA T-score. Our study cohort was divided into two groups: osteoporosis and non-osteoporosis groups, which consist of normal and osteopenia patients. This classification was based on the following two reasons. First, in the Korean medical insurance system, insurance coverage is applied only to patients diagnosed with osteoporosis. Second, the purpose of this study was to evaluate the predicting performance of opportunistic screening for osteoporosis using APCT. Thus, this study was focused on osteoporosis prediction. In the future, it will be necessary to evaluate whether radiomics analysis is possible to predict osteoporosis, osteopenia, and normal status accurately. Our study included a large number of patients with surveillance of breast cancer metastasis. However, in order to minimize the possibility of potential breast cancer metastasis in imaging analysis, we selected the breast cancer patients under the specific evaluation that they were those who did not find any metastases in three consecutive APCT at 6-month intervals. Give that osteoporosis is a common disease in women, the expected effect will be great if the opportunistic screening of osteoporosis is performed simultaneously in female patients diagnosed with breast cancer.

In conclusion, prediction performance of femoral osteoporosis using the machine-learning analysis with radiomics features and APCT proved high validity with more than 93% of accuracy, specificity, and negative predictive value. Overall, opportunistic screening of femoral osteoporosis with machine-learning analysis and APCT has shown high potential feasibility.

Supporting information

S1 Table

(DOCX)

S1 Appendix

(DOCX)

Acknowledgments

The dataset in this paper can be fully accessed through the following address.

“Ha, Hong Il; Lim, Hyun Kyung; Park, Sun-Young; & Han, Junhee (2021). Prediction of femoral osteoporosis using machine-learning analysis with radiomics features and abdomen-pelvic CT: A retrospective single center preliminary study. Zenodo. https://doi.org/10.5281/zenodo.4460972”.

Abbreviations

APCT

abdomen-pelvic CT

AUC

area under the curve

BMD

bone mineral density

CTHU

CT Hounsfield unit

DXA

dual-energy X-ray absorptiometry

HU

Hounsfield unit

HUHA

Hounsfield unit histogram analysis

ICC

intraclass correlation coefficient

RF

random Forest

VOI

volume of interest

Data Availability

The datasets in this paper can be fully accessed through Zenodo: https://doi.org/10.5281/zenodo.4460972.

Funding Statement

This study was supported by the Soonchunhyang University Research fund and the DongKook Life Science. Co., Ltd., Republic of Korea (DK-IIT2019-03). The DongKook Life Science. Co., Ltd. has no competing interests relating to employment, consultancy, patents, products in development, marketed products, etc. These funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

References

  • 1.Sambrook P, Cooper C. Osteoporosis. Lancet. 2006;367(9527):2010–8. 10.1016/S0140-6736(06)68891-0 . [DOI] [PubMed] [Google Scholar]
  • 2.Curtis EM, Moon RJ, Harvey NC, Cooper C. The impact of fragility fracture and approaches to osteoporosis risk assessment worldwide. Bone. 2017;104:29–38. Epub 2017/01/26. 10.1016/j.bone.2017.01.024 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Grimes DA, Schulz KF. Uses and abuses of screening tests. Lancet. 2002;359(9309):881–4. Epub 2002/03/19. 10.1016/S0140-6736(02)07948-5 . [DOI] [PubMed] [Google Scholar]
  • 4.Curtis JR, Carbone L, Cheng H, Hayes B, Laster A, Matthews R, et al. Longitudinal trends in use of bone mass measurement among older americans, 1999–2005. Journal of bone and mineral research: the official journal of the American Society for Bone and Mineral Research. 2008;23(7):1061–7. Epub 2008/02/28. 10.1359/jbmr.080232 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Lim HK, Ha HI, Park SY, Lee K. Comparison of the diagnostic performance of CT Hounsfield unit histogram analysis and dual-energy X-ray absorptiometry in predicting osteoporosis of the femur. Eur Radiol. 2019;29(4):1831–40. Epub 2018/09/27. 10.1007/s00330-018-5728-0 . [DOI] [PubMed] [Google Scholar]
  • 6.Jang S, Graffy PM, Ziemlewicz TJ, Lee SJ, Summers RM, Pickhardt PJ. Opportunistic Osteoporosis Screening at Routine Abdominal and Thoracic CT: Normative L1 Trabecular Attenuation Values in More than 20 000 Adults. Radiology. 2019;291(2):360–7. Epub 2019/03/27. 10.1148/radiol.2019181648 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Alacreu E, Moratal D, Arana E. Opportunistic screening for osteoporosis by routine CT in Southern Europe. Osteoporos Int. 2017;28(3):983–90. Epub 2017/01/22. 10.1007/s00198-016-3804-3 . [DOI] [PubMed] [Google Scholar]
  • 8.Yip SS, Aerts HJ. Applications and limitations of radiomics. Phys Med Biol. 2016;61(13):R150–66. Epub 2016/06/09. 10.1088/0031-9155/61/13/R150 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Lambin P, Rios-Velazquez E, Leijenaar R, Carvalho S, van Stiphout RG, Granton P, et al. Radiomics: extracting more information from medical images using advanced feature analysis. European journal of cancer (Oxford, England: 1990). 2012;48(4):441–6. Epub 2012/01/20. 10.1016/j.ejca.2011.11.036 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Gillies RJ, Kinahan PE, Hricak H. Radiomics: Images Are More than Pictures, They Are Data. Radiology. 2016;278(2):563–77. Epub 2015/11/19. 10.1148/radiol.2015151169 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Fujii M, Aoki T, Okada Y, Mori H, Kinoshita S, Hayashida Y, et al. Prediction of Femoral Neck Strength in Patients with Diabetes Mellitus with Trabecular Bone Analysis and Tomosynthesis Images. Radiology. 2016;281(3):933–9. Epub 2016/06/17. 10.1148/radiol.2016151657 . [DOI] [PubMed] [Google Scholar]
  • 12.Pickhardt PJ, Lee LJ, del Rio AM, Lauder T, Bruce RJ, Summers RM, et al. Simultaneous screening for osteoporosis at CT colonography: bone mineral density assessment using MDCT attenuation techniques compared with the DXA reference standard. Journal of bone and mineral research: the official journal of the American Society for Bone and Mineral Research. 2011;26(9):2194–203. 10.1002/jbmr.428 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Karunanithi R, Ganesan S, Panicker TM, Korath MP, Jagadeesan K. Assessment of bone mineral density by DXA and the trabecular microarchitecture of the calcaneum by texture analysis in pre- and postmenopausal women in the evaluation of osteoporosis. J Med Phys. 2007;32(4):161–8. Epub 2007/10/01. 10.4103/0971-6203.37481 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Abdollahi H, Mahdavi SR, Shiri I, Mofid B, Bakhshandeh M, Rahmani K. Magnetic resonance imaging radiomic feature analysis of radiation-induced femoral head changes in prostate cancer radiotherapy. J Cancer Res Ther. 2019;15(Supplement):S11–S9. Epub 2019/03/23. 10.4103/jcrt.JCRT_172_18 . [DOI] [PubMed] [Google Scholar]
  • 15.Rastegar S, Vaziri M, Qasempour Y, Akhash MR, Abdalvand N, Shiri I, et al. Radiomics for classification of bone mineral loss: A machine learning study. Diagn Interv Imaging. 2020;101(9):599–610. Epub 2020/02/09. 10.1016/j.diii.2020.01.008 . [DOI] [PubMed] [Google Scholar]
  • 16.Obermeyer Z, Emanuel EJ. Predicting the Future—Big Data, Machine Learning, and Clinical Medicine. N Engl J Med. 2016;375(13):1216–9. Epub 2016/09/30. 10.1056/NEJMp1606181 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Pompe E, Willemink MJ, Dijkhuis GR, Verhaar HJ, Mohamed Hoesein FA, de Jong PA. Intravenous contrast injection significantly affects bone mineral density measured on CT. Eur Radiol. 2015;25(2):283–9. 10.1007/s00330-014-3408-2 . [DOI] [PubMed] [Google Scholar]
  • 18.Lambin P, Leijenaar RTH, Deist TM, Peerlings J, de Jong EEC, van Timmeren J, et al. Radiomics: the bridge between medical imaging and personalized medicine. Nat Rev Clin Oncol. 2017;14(12):749–62. Epub 2017/10/05. 10.1038/nrclinonc.2017.141 . [DOI] [PubMed] [Google Scholar]
  • 19.Parekh V, Jacobs MA. Radiomics: a new application from established techniques. Expert Rev Precis Med Drug Dev. 2016;1(2):207–26. Epub 2017/01/04. 10.1080/23808993.2016.1164013 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Depeursinge A, Puspoki Z, Ward JP, Unser M. Steerable Wavelet Machines (SWM): Learning Moving Frames for Texture Classification. IEEE Trans Image Process. 2017;26(4):1626–36. Epub 2017/01/20. 10.1109/TIP.2017.2655438 . [DOI] [PubMed] [Google Scholar]
  • 21.Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD). Ann Intern Med. 2015;162(10):735–6. Epub 2015/05/20. 10.7326/l15-5093-2 . [DOI] [PubMed] [Google Scholar]
  • 22.Kocak B, Durmaz ES, Ates E, Kilickesmez O. Radiomics with artificial intelligence: a practical guide for beginners. Diagn Interv Radiol. 2019;25(6):485–95. Epub 2019/10/28. 10.5152/dir.2019.19321 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Kuhn M. Building Predictive Models in R Using the caret Package. J Stat Softw. 2008;28(5):1–26.27774042 [Google Scholar]
  • 24.King AB, Fiorentino DM. Medicare payment cuts for osteoporosis testing reduced use despite tests’ benefit in reducing fractures. Health Aff (Millwood). 2011;30(12):2362–70. Epub 2011/12/08. 10.1377/hlthaff.2011.0233 . [DOI] [PubMed] [Google Scholar]
  • 25.Raisz LG. Clinical practice. Screening for osteoporosis. N Engl J Med. 2005;353(2):164–71. Epub 2005/07/15. 10.1056/NEJMcp042092 . [DOI] [PubMed] [Google Scholar]
  • 26.Agten CA, Ramme AJ, Kang S, Honig S, Chang G. Cost-effectiveness of Virtual Bone Strength Testing in Osteoporosis Screening Programs for Postmenopausal Women in the United States. Radiology. 2017;285(2):506–17. Epub 2017/06/15. 10.1148/radiol.2017161259 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Crandall CJ, Larson J, Gourlay ML, Donaldson MG, LaCroix A, Cauley JA, et al. Osteoporosis Screening in Postmenopausal Women 50–64 years-old: Comparison of U.S. Preventive Services Task Force Strategy and Two Traditional Strategies in the Women’s Health Initiative. Journal of bone and mineral research: the official journal of the American Society for Bone and Mineral Research. 2014;29(7):1661–6. 10.1002/jbmr.2174 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Nelson HD, Haney EM, Dana T, Bougatsos C, Chou R. Screening for osteoporosis: an update for the U.S. Preventive Services Task Force. Ann Intern Med. 2010;153(2):99–111. Epub 2010/07/14. 10.7326/0003-4819-153-2-201007200-00262 . [DOI] [PubMed] [Google Scholar]
  • 29.Grover M, Anderson M, Gupta R, Haden M, Hartmark-Hill J, Morski LM, et al. Increased osteoporosis screening rates associated with the provision of a preventive health examination. J Am Board Fam Med. 2009;22(6):655–62. Epub 2009/11/10. 10.3122/jabfm.2009.06.090040 . [DOI] [PubMed] [Google Scholar]
  • 30.Ziemlewicz TJ, Binkley N, Pickhardt PJ. Opportunistic Osteoporosis Screening: Addition of Quantitative CT Bone Mineral Density Evaluation to CT Colonography. J Am Coll Radiol. 2015;12(10):1036–41. Epub 2015/10/06. 10.1016/j.jacr.2015.04.018 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Herlidou S, Grebe R, Grados F, Leuyer N, Fardellone P, Meyer ME. Influence of age and osteoporosis on calcaneus trabecular bone structure: a preliminary in vivo MRI study by quantitative texture analysis. Magn Reson Imaging. 2004;22(2):237–43. Epub 2004/03/11. 10.1016/j.mri.2003.07.007 . [DOI] [PubMed] [Google Scholar]
  • 32.Chiba K, Burghardt AJ, Osaki M, Majumdar S. Heterogeneity of bone microstructure in the femoral head in patients with osteoporosis: an ex vivo HR-pQCT study. Bone. 2013;56(1):139–46. Epub 2013/06/12. 10.1016/j.bone.2013.05.019 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Pickhardt PJ, Lee SJ, Liu J, Yao J, Lay N, Graffy PM, et al. Population-based opportunistic osteoporosis screening: Validation of a fully automated CT tool for assessing longitudinal BMD changes. Br J Radiol. 2019;92(1094):20180726. Epub 2018/11/16. 10.1259/bjr.20180726 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Preda L, Lovati E, Chiesa F, Ansarin M, Cattaneo L, Fasani R, et al. Measurement by multidetector CT scan of the volume of hypopharyngeal and laryngeal tumours: accuracy and reproducibility. Eur Radiol. 2007;17(8):2096–102. Epub 2007/04/13. 10.1007/s00330-006-0573-y . [DOI] [PubMed] [Google Scholar]
  • 35.Buerke B, Puesken M, Muter S, Weckesser M, Gerss J, Heindel W, et al. Measurement accuracy and reproducibility of semiautomated metric and volumetric lymph node analysis in MDCT. AJR Am J Roentgenol. 2010;195(4):979–85. Epub 2010/09/23. 10.2214/AJR.10.4010 . [DOI] [PubMed] [Google Scholar]
  • 36.Barrett-Connor E, Siris ES, Wehren LE, Miller PD, Abbott TA, Berger ML, et al. Osteoporosis and fracture risk in women of different ethnic groups. Journal of bone and mineral research: the official journal of the American Society for Bone and Mineral Research. 2005;20(2):185–94. Epub 2005/01/14. 10.1359/JBMR.041007 . [DOI] [PubMed] [Google Scholar]

Decision Letter 0

Alfredo Vellido

19 Jan 2021

PONE-D-20-33039

Prediction of femoral osteoporosis using machine-learning analysis with radiomics features and abdomen-pelvic CT: A retrospective single center preliminary study

PLOS ONE

Dear Dr. Ha,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

We suggest you pay special attention to address the experimental design concerns of reviewer #1.​

Please submit your revised manuscript by Mar 05 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Alfredo Vellido

Academic Editor

PLOS ONE

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please state whether the data utilized in this study were de-identified/anonymised before access?

3. Thank you for stating the following in the Financial Disclosure section:

'Hyun Kyung Lim received the Soonchunhyang University research fund. This fund has no specific grant number.

Hong Il Ha received fund from the DongKook Life Science. Co., Ltd., Republic of Korea (DK-IIT2019-03).

These funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.'

We note that you received funding from a commercial source: DongKook Life Science Co Ltd.

a. Please provide an amended Competing Interests Statement that explicitly states this commercial funder, along with any other relevant declarations relating to employment, consultancy, patents, products in development, marketed products, etc.

Within this Competing Interests Statement, please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials by including the following statement: "This does not alter our adherence to PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests).  If there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

b. Please include your amended Competing Interests Statement within your cover letter. We will change the online submission form on your behalf.

Please know it is PLOS ONE policy for corresponding authors to declare, on behalf of all authors, all potential competing interests for the purposes of transparency. PLOS defines a competing interest as anything that interferes with, or could reasonably be perceived as interfering with, the full and objective presentation, peer review, editorial decision-making, or publication of research or non-research articles submitted to one of the journals. Competing interests can be financial or non-financial, professional, or personal. Competing interests can arise in relationship to an organization or another person. Please follow this link to our website for more details on competing interests: http://journals.plos.org/plosone/s/competing-interests

4. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: " Prediction of femoral osteoporosis using machine-learning analysis with radiomics features and abdomen-pelvic CT: A retrospective single center preliminary study”

It is very interesting that your academic paper evaluated the prediction performance of femoral osteoporosis using machine-learning analysis with radiomics features and APCT. This manuscript is very well written. However, there are some corrections that are essential to meet the standard for publication. Please refer to the following comments.

1. Did you evaluate the validity of the model by cross-validation? If not, please add it. If you have done so, please add it to the supplementary materials.

2. Please indicate the hyperparameters of Random Forest. Did you adjust the hyperparameters in this study? If so, please add the optimized method and result. If you haven't done so, expect to tune for this result.

3. Which data did you use when you took multiple CT scans during the survey period? Also, did any of the same patients have multiple BMD measurements? Please indicate whether the patient's data used either one or both were included in the analysis.

4. In recent years, deep learning has been attracting attention. The authors have shown very good results in machine learning. Make a comparison with deep learning and show your thoughts.

Reviewer #2: Please use the following studies in your paper and compare the results:

1) Radiomics for classification of bone mineral loss: A machine learning study

2) Magnetic resonance imaging radiomic feature analysis of radiation-induced femoral head changes in prostate cancer radiotherapy

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: I do not sign this review on behalf of another person

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2021 Mar 4;16(3):e0247330. doi: 10.1371/journal.pone.0247330.r002

Author response to Decision Letter 0


29 Jan 2021

Point by point response of Reviewer comment

Response to the general comments by editor and editorial office:

1. We revised the financial disclosure section information.

a. Please provide an amended Competing Interests Statement that explicitly states this commercial funder, along with any other relevant declarations relating to employment, consultancy, patents, products in development, marketed products, etc.

� We provide an amended competing interest statement as following,

“The DongKook Life Science. Co., Ltd. has no competing interests relating to employment, consultancy, patents, products in development, marketed products, etc. These funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. These do not alter our adherence to PLOS ONE policies on sharing data and materials.”

b. Please include your amended Competing Interests Statement within your cover letter. We will change the online submission form on your behalf.

� We add the amended Competing Interests Statement within our revised cover letter.

2. We include captions of supporting information files at the end of our manuscript.

Supporting information files

S1. Table of five-fold cross validation results of random forest model.

S2. Appendix of graphical explanation of random forest model.

3. We added data availability statement in the cover letter and manuscript.

Data Availability: The datasets in this paper can be fully accessed through the following address.

“Ha, Hong Il; Lim, Hyun Kyung; Park, Sun-Young; & Han, Junhee (2021). Prediction of femoral osteoporosis using machine-learning analysis with radiomics features and abdomen-pelvic CT: A retrospective single center preliminary study. Zenodo. https://doi.org/10.5281/zenodo.4460972”

Comment and response of Reviewer #1

1. Did you evaluate the validity of the model by cross-validation? If not, please add it. If you have done so, please add it to the supplementary materials.

� We performed 5-fold cross validation and added it to supplementary material named as “S1. Table of five-fold cross validation results of random forest model”.

2. Please indicate the hyperparameters of Random Forest. Did you adjust the hyperparameters in this study? If so, please add the optimized method and result. If you haven't done so, expect to tune for this result.

� We add the tuning of hyperparameters of random forest model as follows,

“For the best hyperparameter tuning, five-fold cross validation using random search was performed and the result was summarized in supplementary file (S1. Table of five-fold cross validation results of random forest model). The hyperparameter of our RF algorithm were as follows: mtry=3, minimum nodal size=11, and splitrule=extratrees. The best model was selected and validated in the test cohort. Further explanation of RF model and features was summarized in supplementary file (S2. Appendix of graphical explanation of random forest model).

3. Which data did you use when you took multiple CT scans during the survey period? Also, did any of the same patients have multiple BMD measurements? Please indicate whether the patient's data used either one or both were included in the analysis.

� No. 500 APCT cases were obtained from 500 non-duplicated patients and all patients had matched BMD test results. To avoid confusion, we add following sentence.

“There were no duplicate patients enrolled.”

4. In recent years, deep learning has been attracting attention. The authors have shown very good results in machine learning. Make a comparison with deep learning and show your thoughts.

� We add following paragraph in discussion section.

“Furthermore, we researched the prediction of osteoporosis by applying machine learning analysis using specific radiomics features and HUHA due to the limited number of patients in this study. Recent AI research is moving to deep learning, which is the evolution of machine learning and it helps in making better precision medicine than machine learning. Deep learning is similar to machine learning, but it does not require artificial intervention. However, it requires big data to train the model otherwise it won’t work as expected. If osteoporosis screening can be implemented by deep learning using wide and easily accessible plain radiographs or large number of CT images in the future, great progress can be expected in the prevention and treatment of osteoporosis.”

Comment and response of Reviewer #2

Please use the following studies in your paper and compare the results:

1) Radiomics for classification of bone mineral loss: A machine learning study

Rastegar S, Vaziri M, Qasempour Y, Akhash MR, Abdalvand N, Shiri I, Abdollahi H, Zaidi H. Radiomics for classification of bone mineral loss: A machine learning study. Diagn Interv Imaging. 2020 Sep;101(9):599-610. doi: 10.1016/j.diii.2020.01.008. Epub 2020 Feb 4. PMID: 32033913.

2) Magnetic resonance imaging radiomic feature analysis of radiation-induced femoral head changes in prostate cancer radiotherapy

Abdollahi H, Mahdavi SR, Shiri I, Mofid B, Bakhshandeh M, Rahmani K. Magnetic resonance imaging radiomic feature analysis of radiation-induced femoral head changes in prostate cancer radiotherapy. J Cancer Res Ther. 2019 Mar;15(Supplement):S11-S19. doi: 10.4103/jcrt.JCRT_172_18. PMID: 30900614.

� We thoroughly reviewed the papers presented. However, we found that the experimental methods and the subject of the study differed greatly from our study. Therefore, the comparison and discussion between the results of these papers and the results of our research does not seem to fit the subject of this research theme. However, since both of these papers are valuable papers that analyzed femoral structural changes through radiomics analysis, we cite these papers as references to the radiomics analysis description mentioned in the introduction as #14 and #15, respectively.

Attachment

Submitted filename: Point by point response of Reviewer comment_plos One.docx

Decision Letter 1

Alfredo Vellido

2 Feb 2021

PONE-D-20-33039R1

Prediction of femoral osteoporosis using machine-learning analysis with radiomics features and abdomen-pelvic CT: A retrospective single center preliminary study

PLOS ONE

Dear Dr. Ha,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Mar 19 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Alfredo Vellido

Academic Editor

PLOS ONE

Additional Editor Comments (if provided):

Dear authors, one of the reviewers is already happy with the current version of your manuscript, but you still need to carefully address the concerns raised by the other reviewer.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: (No Response)

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Thank you for giving me this opportunity to re-review your revised manuscript.

I am happy that almost all of the suggested corrections have been made.

Please refer only to the minor comments below.

Thank you for spending so much time for revised manuscript.

1. Did you evaluate the validity of the model by cross-validation? If not, please add it. If you have done so, please add it to the supplementary materials.

The authors explained that they performed cross-validation to fine-tune the hyperparameters.

Cross-validation is generally an analysis performed to ensure generalization.

Is the result of this analysis of the gini coefficient etc. the holdout method?

Please show the results of cross-validation.

In addition, please show me how to split data for cross-validation. Results may vary depending on the proportion of test data.

Additionally, did you change the hyperparameter adjustment from the default value? Also show the changes from the initial values of hyperparameters.

Reviewer #2: It is an interesting study on the bone radiomics analysis.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2021 Mar 4;16(3):e0247330. doi: 10.1371/journal.pone.0247330.r004

Author response to Decision Letter 1


2 Feb 2021

Response to Reviewers

Reviewer #1: Thank you for giving me this opportunity to re-review your revised manuscript.

I am happy that almost all of the suggested corrections have been made.

Please refer only to the minor comments below.

Thank you for spending so much time for revised manuscript.

1. Did you evaluate the validity of the model by cross-validation? If not, please add it. If you have done so, please add it to the supplementary materials.

The authors explained that they performed cross-validation to fine-tune the hyperparameters.

Cross-validation is generally an analysis performed to ensure generalization.

Is the result of this analysis of the gini coefficient etc. the holdout method?

Please show the results of cross-validation.

In addition, please show me how to split data for cross-validation. Results may vary depending on the proportion of test data.

Additionally, did you change the hyperparameter adjustment from the default value? Also show the changes from the initial values of hyperparameters

� We attach the original results of 5-folds cross validation confusion matrix. Supplementary file named “S1. Table” is summary of this results.

We divided the total 500 cases into 70% of the train set and 30% of the validation set. And hyperparameter tuning was performed while performing 5-fold cross validation on the train set. The test set result was the validation result of RF model using a tuned hyperparameter by 5-fold cross validation.

In addition, we attach the our R-statistical code. This code contains all information of our random forest model about data split, train and test set composition and hyperparameter searching algorithm, etc. This is critical information of this research because we have 2nd phase study related this research. So, we can't publish this "R-code data". However, we already submit our full data set so anyone can test and validate our research.

A. Five-fold cross validation confusion matrix results

1) Fold 1.

2) Fold 2

3) Fold 3

4) fold 4

5) Fold 5

B. R- code for random forest model

library(randomForest)

library(MASS)

library(caret)

library(dplyr)

library(caTools)

library(e1071)

library(tidyverse)

library(tictoc)

library(janitor)

library(doSNOW)

library(ranger)

library(BradleyTerry2)

library(randomForestExplainer)

setwd()

getwd()

data <- read.csv(file="data.csv", header=T)

data["Dx"] <- as.factor(data$Dx)

levels(data$Dx)=c("negative","positive")

set.seed(123)

trainIndex <- createDataPartition(data$Dx, p=.7, list=FALSE, times=1)

train = data[trainIndex,]

test = data[-trainIndex,]

print(table(train$Dx) / nrow(train))

print(table(test$Dx) / nrow(test))

# K-folds

set.seed(123)

cv_folds_lst <- createFolds(train$Dx, k=5, list=FALSE)

set.seed(123)

cv_folds <- createFolds(train$Dx, k=5, list=TRUE)

ranger_tune_grid <- expand.grid(

.mtry = c(2:32),

.splitrule = c("gini","extratrees"),

.min.node.size = c(2:30)

)

fit_ctrl <-trainControl(method = "repeatedcv",

number = 5,

repeats = 5,

index = cv_folds,

summaryFunction=twoClassSummary,

classProbs = TRUE,

verboseIter = TRUE)

set.seed(123)

gc_grid_ranger_model <- train(Dx ~., train,

method = "ranger",

metric = "AUC",

preProcess = c("zv", "center", "scale", "spatialSign"),

tuneGrid = ranger_tune_grid,

#tuneLength = 15,

trControl = fit_ctrl)

gc_grid_ranger_model

# The final values used for the model were mtry = 10, splitrule = extratrees and min.node.size = 3.

fit_ctrl <- trainControl(method = "adaptive_cv",

number = 5,

repeats = 1,

index = cv_folds,

search = "random",

adaptive = list(min = 3, alpha = 0.05, method = "BT", complete = FALSE),

summaryFunction = twoClassSummary,

classProbs = TRUE,

verboseIter = TRUE)

gc_ranger_model <- train(Dx ~., train,

method = "ranger",

metric = "Sens",

preProcess = c("zv", "center", "scale", "spatialSign"),

trControl = fit_ctrl,

tuneLength = 7)

gc_ranger_model

for (ntrees in c(25, 50, 100, 250, 500, 750, 1000, 2000)){

print(ntrees)

acc_vec <- c()

for (idx in c(1, 2, 3, 4, 5)){

num <- numeric(idx)

#print(idx)

cv_train <- train[cv_folds_lst != idx,]

cv_validation <- train[cv_folds_lst == idx,]

X_validation <- cv_validation[, -33]

y_validation <- cv_validation[, 33]

formula.init <- "Dx ~ ."

formula.init <- as.formula(formula.init)

set.seed(123)

rf.model <- randomForest(formula=formula.init, data=cv_train, proximity=T, ntree=ntrees,mtry = 3

, splitrule="extratrees", min.node.size=11)

rf.predictions <- predict(rf.model,X_validation, type="class")

#print(rf.model)

#print(confusionMatrix(data=rf.predictions, reference =y_validation, positive="No"))

acc_vec <- append(acc_vec,mean(rf.predictions == y_validation))

}

print(acc_vec)

print(mean(acc_vec))

}

for (idx in c(1, 2, 3, 4, 5)){

num <- numeric(idx)

print(idx)

cv_train <- train[cv_folds_lst != idx,]

cv_validation <- train[cv_folds_lst == idx,]

X_validation <- cv_validation[, -33]

y_validation <- cv_validation[, 33]

formula.init <- "Dx ~ ."

formula.init <- as.formula(formula.init)

rf.model <- randomForest(formula=formula.init, data=cv_train, proximity=T,ntree=500,mtry = 3,

splitrule="extratrees", min.node.size=11)

rf.predictions <- predict(rf.model,X_validation, type="class")

print(confusionMatrix(data=rf.predictions, reference =y_validation, positive="positive"))

}

X_test <- test[, -33]

y_test <- test[, 33]

formula.init <- "Dx ~ ."

formula.init <- as.formula(formula.init)

set.seed(123)

rf.model <- randomForest(formula=formula.init, data=train, proximity=T,ntree=500,mtry =3, splitrule="extratrees", min.node.size=11)

rf.predictions <- predict(rf.model,X_test, type="class")

confusion <- confusionMatrix(data=rf.predictions, reference =y_test, positive="?缺")

rf.model$importance

varImpPlot(rf.model)

X_test <- test[, -33]

y_test <- test[, 33]

test_predict <- predict(rf.model,X_test, type="class")

levels(y_test)=c(0,1)

levels(test_predict)=c(0,1)

y_test <- as.numeric(levels(y_test))[y_test]

test_predict <- as.numeric(levels(test_predict))[test_predict]

train_model <- rf.model$predicted

train_model <- as.factor(as.numeric(train_model))

levels(train_model)=c(0,1)

library(pROC)

par(pty="s")

trainROC <- roc(y_train ~ as.numeric(levels(train_model))[train_model],plot=TRUE,print.auc=TRUE,col="blue",

lwd =4,legacy.axes=TRUE,main="ROC Curves", percent=TRUE)

#ylab="False Positive Percentage", xlab = "True Positive Percentage"

## Setting levels: control = 0, case = 1

## Setting direction: controls < cases

testROC <- roc(y_test ~ test_predict,plot=TRUE,print.auc=TRUE,col="red",lwd = 4,print.auc.y=45,legacy.axes=TRUE,add = TRUE, percent=TRUE)

## Setting levels: control = 0, case = 1

## Setting direction: controls < cases

legend("bottomright",legend=c("Training set","Test set"),col=c("blue","red"),lwd=4)

library(ROCR)

library(pROC)

library(randomForest)

#data dependent variable set

set.seed(123)

train$Dx = as.factor(train$Dx)

data1.rf <- randomForest(Dx ~., data=train, proximity=T,ntree=500,mtry = 3, splitrule="extratrees", min.node.size=11)

test$Dx = as.factor(test$Dx)

data2.rf <- randomForest(Dx ~., data=test,proximity=T,ntree=500,mtry = 3, splitrule="extratrees", min.node.size=11)

par(pty="s")

set.seed(123)

require(pROC)

rf.roc1 <-roc(train$Dx,data1.rf$votes[,2], plot=TRUE, legacy.axes=TRUE, percent = TRUE,

xlab="1-Specificity",ylab='Sensitivity',col='#FF0000',

lwd=4,

print.auc=TRUE,print.auc.y=25)

rf.roc2 <-roc(test$Dx,data2.rf$votes[,2], plot=TRUE, legacy.axes=TRUE, percent = TRUE,

xlab="1-Specificity",ylab='Sensitivity',col='#0000FF',

lwd=4,

print.auc=TRUE,add=TRUE, print.auc.y=20)

ci(rf.roc1)

ci(rf.roc2)

roc.test(rf.roc1, rf.roc2, method=c("delong", "bootstrap",

"venkatraman", "sensitivity", "specificity"), sensitivity = NULL,

specificity = NULL, alternative = c("two.sided", "less", "greater"),

paired=NULL, reuse.auc=TRUE, boot.n=2000, boot.stratified=TRUE,

ties.method="first", progress=getOption("pROCProgress")$name,

parallel=FALSE)

library(randomForestExplainer)

library(randomForest)

library(tidyverse)

set.seed(123)

forest <- randomForest::randomForest(Dx ~ ., data = data, localImp = TRUE, proximity=T,mtry = 3, splitrule="extratrees", min.node.size=11)

suppressPackageStartupMessages(suppressMessages(suppressWarnings(explain_forest(forest, interactions = TRUE))))

Attachment

Submitted filename: point by point response of Reviewer comment2 Plos one.docx

Decision Letter 2

Alfredo Vellido

5 Feb 2021

Prediction of femoral osteoporosis using machine-learning analysis with radiomics features and abdomen-pelvic CT: A retrospective single center preliminary study

PONE-D-20-33039R2

Dear Dr. Ha,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Alfredo Vellido

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Thank you for giving me this opportunity to re-review your revised manuscript.

I am happy that all of the suggested corrections have been made.

Thank you for spending so much effort.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Acceptance letter

Alfredo Vellido

10 Feb 2021

PONE-D-20-33039R2

Prediction of femoral osteoporosis using machine-learning analysis with radiomics features and abdomen-pelvic CT: A retrospective single center preliminary study

Dear Dr. Ha:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Alfredo Vellido

Academic Editor

PLOS ONE

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Supplementary Materials

    S1 Table

    (DOCX)

    S1 Appendix

    (DOCX)

    Attachment

    Submitted filename: Point by point response of Reviewer comment_plos One.docx

    Attachment

    Submitted filename: point by point response of Reviewer comment2 Plos one.docx

    Data Availability Statement

    The datasets in this paper can be fully accessed through Zenodo: https://doi.org/10.5281/zenodo.4460972.


    Articles from PLoS ONE are provided here courtesy of PLOS

    RESOURCES