Predictive value of machine learning on fracture risk in osteoporosis: a systematic review and meta-analysis

Abstract

Objectives

Early identification of fracture risk in patients with osteoporosis is essential. Machine learning (ML) has emerged as a promising technique to predict the risk, whereas its predictive performance remains controversial. Therefore, we conducted this systematic review and meta-analysis to explore the predictive efficiency of ML for the risk of fracture in patients with osteoporosis.

Methods

Relevant studies were retrieved from four databases (PubMed, Embase, Cochrane Library and Web of Science) until 31 May 2023. A meta-analysis of the C-index was performed using a random-effects model, while a bivariate mixed-effects model was used for the meta-analysis of sensitivity and specificity. In addition, subgroup analysis was performed according to the types of ML models and fracture sites.

Results

Fifty-three studies were included in our meta-analysis, involving 15 209 268 patients, 86 prediction models specifically developed for the osteoporosis population and 41 validation sets. The most commonly used predictors in these models encompassed age, BMI, past fracture history, bone mineral density T-score, history of falls, BMD, radiomics data, weight, height, gender and other chronic diseases. Overall, the pooled C-index of ML was 0.75 (95% CI: 0.72, 0.78) and 0.75 (95% CI: 0.71, 0.78) in the training set and validation set, respectively; the pooled sensitivity was 0.79 (95% CI: 0.72, 0.84) and 0.76 (95% CI: 0.80, 0.81) in the training set and validation set, respectively; and the pooled specificity was 0.81 (95% CI: 0.75, 0.86) and 0.83 (95% CI: 0.72, 0.90) in the training set and validation set, respectively.

Conclusions

ML has a favourable predictive performance for fracture risk in patients with osteoporosis. However, most current studies lack external validation. Thus, external validation is required to verify the reliability of ML models.

PROSPERO registration number

CRD42022346896.

Strengths and limitations of this study.

• The latest systematic review and meta-analysis conducted to assess machine learning (ML) models for fracture risk.

• We performed a quantitative synthesis to enhance the comparability of ML models.

• C-index, sensitivity and specificity were performed to evaluate the performance of ML models.

• Several studies were included in the systematic review but excluded from subsequent meta-analyses.

• Most of the included studies lack external validation.

Introduction

Osteoporosis is a systemic metabolic bone disease characterised by decreased bone mass and degraded bone microarchitecture, leading to an increased risk of bone fragility fracture.¹ Due to high disability and morbidity rates, high treatment costs and low quality of life of patients, it has emerged as a global health concern.² According to the WHO, osteoporosis is the second most serious health issue after cardiovascular diseases.³ This condition may cause fragility fractures that commonly occur in the wrist, spine and hip. Spine and hip fractures may lead to disability, which not only affects the quality of life and longevity of patients but also causes enormous medical expenses and a heavy burden of care.^{4 5}

Machine learning (ML), a subfield of artificial intelligence, enables computers to ‘learn’ through programmes. Compared with traditional statistical methods, ML emphasises more on the accuracy of prediction and can detect regularities in multi-dimensional data sets. ML algorithms can be basically divided into supervised learning and unsupervised learning.⁶ ML has been applied in the field of osteoporosis, providing a novel method for the prediction of fracture risk. A review by Ferizi et al ⁷ (2019) summarised relevant studies on the application of artificial intelligence to the prediction of osteoporosis. It drew a conclusion that ML methods for automatic image segmentation and fracture risk prediction showed a promising clinical value. A systematic review by Smets et al ⁸ (2021) reviewed the state-of-the-art ML methods and their application in osteoporosis diagnosis and fracture prediction. Another review by Anam et al ⁹ (2021) explored the prediction performance of MRI for osteoporosis in trabecular bone from a methodology-driven and application perspective. Most studies focused on the role of ML in the prediction of osteoporosis indicators, such as bone mineral density (BMD), or in the automatic segmentation of the images of patients at risk of osteoporosis. However, the efficiency of ML in predicting osteoporotic fractures is understudied.

The present study evaluated the predictive performance of ML for fracture risk in osteoporosis patients, providing an evidence-based medical basis for the application of ML in clinical practice.

Materials and methods

This study was conducted in accordance with the Preferred Items of Systematic Review and Meta-Analysis (PRISMA) statement (online supplemental table S1).¹⁰ The protocol was registered on the international prospective register of systematic reviews (PROSPERO) (Registration No. CRD42022346896). Relevant studies were retrieved from Pubmed, Embase, Cochrane Library and Web of Science, and the retrieval was as of 31 May 2023. Two researchers independently searched the literature. The search strategy is shown in online supplemental table S2.

Inclusion criteria were as follows: (1) patients were diagnosed with osteoporosis; (2) ML was applied to predict fracture risk; (3) at least one measure of model performance (discrimination or calibration) was reported; (4) study population included adult patients older than 18 years, mainly including adults, older people and postmenopausal women. Exclusion criteria were as follows: (1) studies that only analysed risk factors without building complete ML models; (2) studies that only included osteoporosis but did not mention fracture risk; (3) studies without available full text (or only abstract available) or data; (4) meta-analyses, reviews, case reports, editorial materials, letters, protocols, errata, and notes.

Two researchers independently extracted data using standardised tables. Any studies excluded after full-text review have been recorded with reasons for their exclusion. The list of extracted items was based on the CHARMS checklist,¹¹ and two data extraction sheets were prepared for developed and validated models, respectively. Finally, the extracted data included the first author, year of publication, country, study design, data source, population group, gender, age, fracture sites, types of predictive models, number of predictors and outcomes. The risk of bias was assessed using the Prediction Model Risk of Bias Assessment Tool (PROBAST). The PROBAST contained a large number of questions in four distinct domains: participants, predictors, outcomes, and statistical analysis, reflecting the overall risk of bias and applicability.¹²

Meta-analysis of C-index, sensitivity and specificity was performed to evaluate the performance of ML models. If the C-index did not report 95% CIs and SEs, we estimated the SEs in reference to the study by Debray et al.¹³ A C-index of 0.5 indicates low discrimination; 0.6 to 0.7 indicates modest discrimination; 0.71 to 0.8 indicates very good discrimination; and greater than 0.8 indicates strong discrimination.¹⁴ When original studies did not report the accuracy, we calculated it based on the sensitivity, specificity, the number of samples in each subgroup and the number of modelling samples.¹³ Given the differences in variables, ML algorithms and parameters across the studies, the random-effects model was preferred for the meta-analysis of C-index, and the bivariate mixed-effects model was used for the meta-analysis of sensitivity and specificity. Heterogeneity was quantified using I ² statistics. Sensitivity analysis was performed to further identify the source of heterogeneity by removing each study and re-calculating the pooled effect size of the remaining studies. The meta-analysis was performed using the software Stata V.15.1 (Stata Corporation) and R V.4.2.0 (R Development Core Team, Vienna, http://www.R-project.org). A p value less than 0.05 was considered statistically significant.

Patient and public involvement

No patients involved.

Results

A total of 12 468 studies were searched from the databases, including 2409 from PubMed, 4387 from Embase, 170 from Cochrane Library and 5502 from Web of Science. After removing duplicates and screening titles and abstracts, 378 articles remained. According to a full-text review, 53 articles^13–67 were included. Fifty-three articles presented the development of one or more prediction models for osteoporotic fracture, while 26 articles described the validation of one or more models. The search process is shown in figure 1.

Figure 1.

The flow chart of retrieval process.

Fifty-three studies were ultimately included in our meta-analysis, involving 15 209 268 patients. Many studies originated from USA (n=11), European(n=11) and China (n=8). Most studies were cohort studies (n=46), and the rest were case–control studies (n=7). The median age of osteoporosis patients was 68.8 years (ranging from 48.5 to 84 years). The study population in most studies covered women (n=24). The fracture sites included multi-site (n=26), vertebra (n=14), hip (n=12) and femur (n=1). Most studies were based on clinical hospital data (n=19), while some used questionnaire collection data (n=10), osteoporosis registry data (n=9), electronic health records (n=7) and administrative data (n=6). Only 13 articles elucidated the cross-validation method. The baseline characteristics of the included studies are shown in online supplemental table S3.

There were 86 prediction models specifically developed for the osteoporosis population and 41 validation sets. Ninety-eight ML models reported the C-index or the area under the receiver operating characteristic curve (AUC), ranging from 0.50 to 0.98. Online supplemental table S4 shows all studies on the development and validation of ML models for outcome prediction in patients with osteoporosis. Among all the identified prediction models, the logistic regression (31.4%) was the most commonly used algorithm, followed by the survival model (18%).

The most commonly used predictors in ML models were age (n=72), body mass index (BMI) (n=40), past fracture history (n=35), BMD T-score (n=33), history of falls (n=29), BMD (n=28), radiomics data (n=25), weight (n=24), height (n=23), gender (n=20), and other chronic diseases (n=20) (table 1).

Table 1.

Main sorts of predictors included in developed models for osteoporosis patients

Predictors	Number of models
Demographics
Age	72
History of falls	29
Sex	20
Women’s menopause age	8
Family genetic history	6
Race	5
Physical examination
Body mass index	40
BMD t-score	33
BMD	28
Weight	24
Height	23
Motor ability	10
Lifestyle
Alcohol consumption	13
Smoking	11
Physical activity	10
Lack of physical exercise	7
Daily activities	5
Limited physical activity	4
Frequent sun exposure	3
Comorbidity
Past fracture history	35
Other chronic diseases	20
Osteoporosis	8
Rheumatoid arthritis	7
Genetic risk score	5
Fracture type	4
Backache	2
Drug and nutrient intake
Use of hormonal drugs	8
Calcium intake	8
Nutritional status	6
Intake of other drugs	4
Radiomics
Radiomic data	25
Mental state
Cognitive performance	3
Anxiety/depression	2

Note: BMD (g/cm²).

BMD, bone mineral density.

The risk of bias assessment of the included studies is summarised in figure 2. More than half of these studies had a high risk of bias (67%). The risk of bias in most studies was low in terms of participants, predictors and outcome. However, a high or unclear risk of bias in the statistical analysis was observed in all studies. More details are shown in online supplemental table S5.

Figure 2.

Risk of bias assessment (using Prediction Model Risk of Bias Assessment Tool) based on four domains across all machine learning models.

Sixty-six training datasets and 32 validation datasets were included in the meta-analysis of the C-index. Since substantial heterogeneity was present, we performed subgroup analyses based on fracture site and model type. Table 2 shows the results of the meta-analysis of C-index of ML models in predicting osteoporosis. Logistic regression is the most widely used method. The forest plot of C-index is presented in online supplemental figures S1 and S2. The pooled C-index was 0.75 (95% CI: 0.72, 0.78) (I ²=99.7%, p<0.001) in the training set and 0.75 (95% CI: 0.71, 0.78) (I ²=99.8%, p<0.001) in the validation set. In the training set, other deep learning method showed the highest predictive performance (pooled C-index=0.97), followed by convolutional neural network (CNN) (pooled C-index=0.94), decision trees (pooled C-index=0.78) and logistic regression (pooled C-index=0.75). Furthermore, models for vertebral fracture (pooled C-index=0.80) and hip fracture (pooled C-index=0.76) outperformed those for multi-site fracture (pooled C-index=0.70). However, in the validation set, CNN (pooled C-index=0.98) showed the best performance, closely followed by other deep learning method (pooled C-index=0.82), logistic regression (pooled C-index=0.80) and support vector machines (SVMs) (pooled C-index=0.78). Models for vertebral fracture (pooled C-index=0.87) outperformed those for hip fracture (pooled C-index=0.73) and multi-site fracture (pooled C-index=0.71). Across these studies, we extracted 57 estimates of balanced accuracy (the average of the reported sensitivity and specificity), ranging from 0.46 to 1.00. As presented in table 3, the mean sensitivity and specificity of models were 0.79 (95% CI: 0.72, 0.84) (I ²=99.2%, p<0.001) and 0.81 (95% CI: 0.75, 0.86) in the training set (I ²=99.9%, p<0.001) and 0.76 (95% CI: 0.80, 0.81) (I ²=98.9%, p<0.001) and 0.83 (95% CI: 0.72, 0.90) in the validation set (I ²=99.9%, p<0.001), respectively. The results of sensitivity analysis show that ML models built for different fracture sites have stable performance in the training and validation sets (online supplemental figures S3–S8).

Table 2.

Results of subgroup analysis of C-index by fracture site and machine learning type

Subgroup	Training dataset		Validation dataset
	N	C-statistic (95% CI)	N	C-statistic (95% CI)
Fracture site
Vertebral fracture	15	0.80 (0.74, 0.87)	6	0.87 (0.71, 1.00)
Hip fracture	20	0.76 (0.72, 0.81)	9	0.73 (0.65, 0.81)
Multi-site fracture	31	0.70 (0.67, 0.72)	17	0.71 (0.65, 0.76)
Model type
LR	26	0.75 (0.72, 0.78)	7	0.80 (0.73, 0.87)
ANN	4	0.73 (0.64, 0.82)	3	0.66 (0.62, 0.70)
CNN	2	0.95 (0.94, 0.96)	1	0.98 (0.94, 1.00)
RF	3	0.70 (0.68, 0.72)	3	0.66 (0.59, 0.73)
SVM	5	0.72 (0.60, 0.85)	3	0.78 (0.59, 0.96)
DT	2	0.78 (0.56, 0.99)	1	0.69 (0.67, 0.70)
NB	2	0.74 (0.39, 1.00)	–
kNN	1	0.51 (0.46, 0.55)	–
Survival model	13	0.70 (0.69, 0.74)	9	0.68 (0.67, 0.69)
Boosted tree	5	0.71 (0.69, 0.74)	3	0.70 (0.69, 0.71)
Ensemble learning	1	0.72 (0.71, 0.73)
Other DL	2	0.97 (0.96, 0.97)	1	0.82 (0.77, 0.87)
Overall	66	0.75 (0.72, 0.78）	32	0.75 (0.71, 0.78)

ANN, artificial neural network; CNN, convolutional neural network; DL, deep learnimg model; DT, decision tree; kNN, k-nearest neighbour; LR, logistic regression; NB, Naive Bayes; RF, random forests; SVM, support vector machine.

Table 3.

Results of subgroup analysis of sensitivity and specificity by fracture site and machine learning type

Subgroup	Training dataset			Validation dataset
	N	Sensitivity (95% CI)	Specificity (95% CI)	N	Sensitivity (95% CI)	Specificity (95% CI)
Fracture site
Vertebral fracture	10	0.73 (0.61, 0.82)	0.91 (0.86, 0.95)	3	0.87 (0.70, 0.95)	0.97 (0.94, 0.98)
Hip fracture	13	0.90 (0.82, 0.94）	0.82 (0.75, 0.88)	5	0.84 (0.77, 0.89)	0.85 (0.80, 0.89)
Multi-site fracture	18	0.71 (0.59, 0.81)	0.72 (0.60, 0.81)	8	0.66 (0.61, 0.70)	0.69 (0.53, 0.81)
Model type
LR	17	0.70 (0.63, 0.77)	0.73 (0.67, 0.79)	4	0.66 (0.55, 0.75)	0.65 (0.50, 0.77)
ANN	4	0.91 (0.70, 0.98)	0.93 (0.75, 0.98)	3	0.78 (0.71, 0.83)	0.85 (0.71, 0.93)
CNN	3	0.83 (0.81, 0.84)	0.91 (0.79, 0.96)	1	0.98	0.95
RF	1	0.84	0.91	1	0.70	0.46
SVM	6	0.81 (0.63, 0.92)	0.63 (0.13, 0.95)	3	0.79 (0.72, 0.85)	0.89 (0.79, 0.94)
DT	2	0.97 (0.53, 1.00)	0.70 (0.67, 0.73)	–
NB	2	0.63 (0.13, 0.95)	0.76 (0.70, 0.81）	–
kNN	2	0.95 (0.39, 1.00）	0.80 (0.77, 0.83）	1	0.81	0.79
Survival model	1	0.81	0.52	–
Boosted tree	1	0.59	0.67	1	0.70	0.95
Other DL	2	0.81 (0.72, 0.87)	0.96 (0.93, 0.98)	2	0.83 (0.74, 0.90)	0.95 (0.92, 0.97)
Overall	41	0.79 (0.72, 0.84)	0.81 (0.75, 0.86)	16	0.76 (0.80, 0.81)	0.83 (0.72, 0.90)

ANN, artificial neural network; CNN, convolutional neural network; DL, deep learnimg model; DT, decision tree; kNN, k-nearest neighbour; LR, logistic regression; NB, Naive Bayes; RF, random forests; SVM, support vector machine.

Discussion

ML is a popular research method that provides new tools for early detection of diseases. This study systematically explored the application of the latest ML methods in predicting fracture risk in osteoporosis. The most commonly used predictors in ML models are age, BMI, past fracture history, BMD T-score, history of falls, BMD, radiomics data, weight, height, gender and other chronic diseases. In general, most predictors included in model development studies are traditional risk factors. A recent study showed that the most common risk factors for fragility fractures encompassed decreased BMD, age, gender, low BMI, history of fragility fractures, family history of hip fractures, history of glucocorticoid therapy, smoking, excessive alcohol consumption, lack of vitamin D, early menopause and immobility.⁶⁸ This is consistent with some common fracture predictors identified in our study. Our study also finds that radiomics data are frequently used as a fracture predictor in ML models for osteoporosis. A retrospective, single-centre, preliminary investigation by Lim et al ⁶⁹ reported that ML based on radiomics features and abdomen–pelvic CT for diagnosing osteoporosis showed high predictive performance, with accuracy, specificity, and negative predictive value exceeding 93%.

The present study found that ML methods commonly used in the field of osteoporosis included logistic regression, decision tree, random forest, survival model, SVM, ensemble learning, artificial neural network (ANN), CNN and the latest deep learning technology. ML has a good performance in the prediction and identification of osteoporosis and fracture. In terms of the models in the training sets, the prediction efficiency of other deep learning method is optimal, followed by CNN, decision trees and logistic regression. In the validation sets, CNN showed the best performance, closely followed by other deep learning method, logistic regression and SVM. Deep learning is more powerful than traditional ML algorithms, with a wide range of coverage. Its performance increases with the amount of data.⁷⁰ Deep learning has been successfully applied to assist in the diagnosis and prediction of osteoporotic fractures.^{34 63} CNN, a core algorithm of deep learning, is widely used in the field of data analysis and disease prediction with high accuracy.⁷¹ CNN techniques can effectively predict the risk of osteoporotic fractures, enabling clinicians to take timely treatment measures, thereby reducing the occurrence of fractures.^{19 23 47} Additionally, logistic regression is an efficient, simple and easy-to-operate ML method that outputs calibrated predicted probabilities. An article on prediction models for the outcomes in patients with chronic obstructive pulmonary disease revealed that logistic regression was the most frequently used modelling method.⁷² This is the same as recent findings reported by Silva et al.⁷³ Their ML models based on logistic regression outperformed those based on random forest and decision trees. Moreover, SVM adapts well to small samples and high-dimensional data with a low misclassification rate, and therefore can be used for classification and regression analysis.⁷⁴

Most included studies report multiple outcomes, such as sensitivity, specificity, AUC and ROC. The mean sensitivity of the models in the training set model is 0.79 (95% CI: 0.72, 0.84) greater than that of the models in the validation set. Most models are internally validated in the same population database and lack external validations in other populations. Only ML models in six articles were externally validated.32 34 36 48 55 67 Therefore, external validations of ML models for predicting fracture risk are needed. However, a single performance measure such as AUC or ROC is insufficient to recommend the application of ML models into clinical practice,⁸ and multiple measures of performance should be combined.

This systematic review and meta-analysis summarised a large number of studies to comprehensively evaluate the predictive performance of ML for fracture risk in patients with osteoporosis. The characteristics of the established and validated models were described. We performed a quantitative synthesis that was never done in previous studies to compare these models. Furthermore, the meta-analysis of C-index was performed using the random-effects model, since the C-index was reported in most predictive models.^{72 75} Meanwhile, the bivariate mixed-effects model was used for the meta-analysis of sensitivity and specificity. In the training dataset, the sensitivity of hip fracture was the highest, closely followed by multi-site fracture and vertebral fracture. For patients with hip fractures, radiographs may cause missed diagnosis and misdiagnosis, leading to poor prognosis.⁷⁶ ML models have been increasingly used to identify hip fracture risk with high accuracy.³¹ ML has a stronger power to recognise images and can assist inexperienced clinicians in a highly accurate diagnosis.

Some limitations still need to be considered in the present study. Due to incomplete reporting of indicators, several studies were only included in the systematic review and were excluded from subsequent meta-analyses.^{61 63} Studies conducted in either Western or Asian populations lack external validation, and thus external validations in other populations are needed to widen the application of ML models. The risk of bias assessment demonstrated that most studies (67%) had a high risk of bias, regardless of whether they involved the development or external validation of a prediction model for the osteoporosis population. The main bias came from the statistical analysis because most studies did not properly handle continuous and categorical variables and reported no method for processing missing values. Only three articles reported the use of median imputation or multiple interpolation method to deal with missing values,^{15 31 66} while others did not mention how to deal with missing values. These shortcomings in the methodology may be due to a lack of guidelines for the standard reporting of risk prediction studies at that time. In addition, some models were reported with little information, making it unable for other researchers to perform external validation, much less the application in clinical practice. For example, only 12 articles used the K-fold cross-validation method to improve the accuracy of their algorithms,15 16 19 27 30 31 34 36 42 46 47 62 but most of the eligible articles did not. Models without stringent validation cannot be widely applied.⁷³ Many studies have limited applicability in clinical practice because of flawed methodologies or unrepresentative data sets. Future research should give priority to the development of practical algorithms. Furthermore, we observed large heterogeneity in the meta-analysis of C-statistics. Potential sources of heterogeneity may be the differences in patients’ characteristics, data sources and analysis methods across the studies. More than 30% of the research data came from clinical studies, and clinical data are heterogeneous and usually imbalanced. At last, most ML models did not report balanced accuracy and lacked calibration or external validation or decision curves. Thus, further research is required to address these issues, improving the generalisation of the models.

Despite the limitations mentioned above, the present study can still provide meaningful recommendations for future research and practice. First, the major strength of our study is the rigorous literature search and methodology to provide reliable estimates. This is the latest systematic review and meta-analysis conducted to comprehensively assess ML models for fracture risk. Second, ML models can provide convincing evidence to assist clinicians in making more accurate judgments during highly complex decision-making processes, with certain clinical application values.⁷⁴ More rigorous, robust and comprehensive research is warranted to assess its clinical application and impact on clinicians and patients. Third, the advances in emerging technologies such as ML have opened a new era of clinical medical research, providing new directions for solving intricate problems with classical statistical methods. However, clinicians currently are not skillful in using such emerging technologies. Therefore, it is advisable for clinicians to improve their ability to use ML to make more accurate diagnoses.

In conclusion, ML has a favourable predictive performance for fracture risk in patients with osteoporosis and can be used as a potential tool for early identification of fracture risk in this population. However, most current studies lack external validation. Therefore, future research is needed to validate and improve the existing predictive models for osteoporosis risk rather than developing new models.

Data availability statement

Data are available upon reasonable request. Data may be obtained from a third party and are not publicly available.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

Not applicable.