Development and validation of a machine learning-based prediction model for frailty in older adults with diabetes: a study protocol for a retrospective cohort study

Abstract

Introduction

Frailty is a common condition in older adults with diabetes, which significantly increases the risk of adverse health outcomes. Early identification of frailty in this population is crucial for implementing timely interventions. However, there is a lack of specific prediction models for frailty in older adults with diabetes. This study aims to develop and validate a prediction model for frailty in this high-risk group.

Methods and analysis

This study uses data from the national follow-up of the China Health and Retirement Longitudinal Study (CHARLS), which range from 2011 to 2020. The study population includes older adults with diabetes aged 60 and above. Frailty is assessed using Fried’s frailty phenotype. Potential predictors will be identified through a systematic review and expert consultation. Eight machine learning models will be developed to predict frailty, with model performance to be evaluated using receiver operating characteristic curves, calibration plots and internal validation through leave-one-out cross validation. Finally, the optimal model will be deployed via an electronic risk calculator with Shapley Additive Explanation-based visualisations.

Ethics and dissemination

The CHARLS was approved by the Biomedical Ethics Committee of Peking University (approval number: IRB00001052-11015), and all participants were required to sign informed consent. This study was approved by the Medical Research Ethics Committee of the University of South China (approval number: 2023NHHL006). We will disseminate results via presentations at scientific meetings and publication in peer-reviewed journals.

PROSPERO registration number

CRD42023470933.

Strengths and limitations of this study.

• The protocol follows the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis—Artificial Intelligence reporting guideline to ensure methodological transparency.

• A predefined statistical analysis plan includes multiple machine learning methods for model development.

• Predictor selection is informed by a systematic review and expert consultation.

• The retrospective design may result in missing data and limit the range of available predictors.

• The black-box nature of machine learning may limit transparency despite Shapley Additive Explanation-based interpretability tools.

Introduction

Diabetes continues to impose a growing health burden worldwide, especially in ageing populations. The 10th edition of the IDF Diabetes Atlas estimates that nearly one in four adults aged 75–79 will have diabetes by 2045, corresponding to a projected prevalence of 24.7%.¹ With ongoing population ageing worldwide, the proportion of adults aged over 60 living with diabetes is projected to rise continuously.² Frailty is increasingly being identified as the third major complication of diabetes, following microvascular and macrovascular complications.³ Frailty is a multidimensional condition marked by decreased physiological reserve and increased sensitivity to stressors due to impairments in multiple systems.⁴ Among older adults with diabetes, frailty is widely recognised as being linked to elevated risks of both mortality and disability.⁵ Importantly, the prevalence of frailty is 3–5 times higher in older adults with diabetes than in their non-diabetic counterparts.⁶ Current research suggests a reciprocal interaction between diabetes and frailty, which may contribute to a cycle of worsening health. Frailty plays a critical role in shaping the prognosis of older adults with diabetes.⁷ Frailty, while commonly linked to ageing, can arise independently and may not always progress over time.⁸ In line with this, evidence indicates that frailty-related impairments in older adults are potentially modifiable and may fluctuate over time.⁹ A tool that can detect and predict frailty at an early stage would significantly enhance healthcare professionals’ decision-making capabilities and resource management.

Machine learning (ML) is a field of computer science that leverages existing data to predict future outcomes when new data are presented.¹⁰ With growing access to data and the availability of off-the-shelf software for applying ML methods, developing prediction models has become faster and more straightforward.¹¹ Numerous studies have been dedicated to the development of frailty prediction models, with researchers employing a variety of ML algorithms to detect, classify and predict the onset of frailty.12,15 These algorithms, including random forests, support vector machines, logistic regression and artificial neural networks, have shown promise in identifying risk factors and patterns that precede the development of frailty. Despite the advancements in this field, the application of these models to older adults with diabetes remains limited. This population is characterised by a complex interplay of metabolic dysregulation, chronic inflammation and multiple comorbidities,¹⁶ which may confound the prediction of frailty. Therefore, a tailored prediction model specifically designed for this subgroup is essential to address the unique challenges faced by older adults with diabetes. This study aims to address this gap by creating a predictive model that combines the complexities of diabetes-related pathophysiology with the phenotypic characteristics of frailty. By doing so, we aim to provide clinicians with a robust tool for the early identification of frailty, enabling personalised care plans that mitigate its negative impacts on the health and well-being of older adults with diabetes.

Objectives

Primary objectives

The primary objective is to develop and validate a predictive model for frailty in older adults with diabetes using data from the 2011 to 2020 China Health and Retirement Longitudinal Study (CHARLS), enabling early identification and timely interventions to improve health outcomes.

Secondary objectives

The secondary objectives are to identify key predictors through systematic review and expert consultation to ensure the model’s relevance, to compare the performance of eight advanced ML algorithms to select the most effective model and to enhance clinical applicability by developing an electronic risk calculator with Shapley Additive Explanation (SHAP)-based visualisations to support interpretation and decision-making in clinical practice.

Methods

We will conduct a prediction model development and validation study using a retrospective cohort design (figure 1). This study will be reported following the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis—Artificial Intelligence (TRIPOD-AI) statement.¹¹

Figure 1. The flowchart of developing the prediction model. CHARLS, China Health and Retirement Longitudinal Study.

Source of data

The research data source is the CHARLS led by Peking University from 2011 to 2020. In the baseline survey (2011), CHARLS surveyed 17 000 middle-aged and older adults aged 45 and above in about 10 000 households.¹⁷ The participants were interviewed face-to-face in their homes using computer-assisted personal interviewing technology. In terms of survey content, the CHARLS questionnaire covers individual and family information of middle-aged and older adults, including physical and mental health status represented by depressive symptoms, chronic disease status and self-rated health, as well as demographic variables such as gender, age, marital status and education level. CHARLS provides the data for this study to conduct a prediction model development and validation for frailty in older adults with diabetes. The CHARLS datasets are available for download from the CHARLS homepage: http://charls.pku.edu.cn/en.

Participants

To be eligible to take part in the study, participants must meet the following criteria: (a) aged ≥60 years (CHARLS variable: ba004_w3_1≤1951) and (b) diagnosed with diabetes,¹⁸ defined as meeting at least one of the following criteria: fasting plasma glucose ≥126 mg/dL (CHARLS variable: newglu), HbA1c ≥6.5% (CHARLS variable: newhba1c) or self-reported physician diagnosis of diabetes (CHARLS variable: DA007). Exclusion criteria: (a) abnormal data, defined as biologically implausible or logically inconsistent values; (b) data missing for frailty items in the CHARLS >10% and (c) data missing for predictor items in the CHARLS >10%. We plan to use the training set (patients from CHARLS 2011 and 2015) to develop the frailty prediction models and use other waves for validation.

Study period

This study is scheduled to be conducted from 1 July 2025 to 1 May 2026, including systematic review and expert consultation (July–September 2025), model development and validation (October 2025–January 2026) and manuscript preparation (February–May 2026).

Outcome

We use the frailty phenotype (FP) developed by Fried¹⁹ to assess frailty status. Currently, FP is the most widely used frailty scale and has been validated in other studies using CHARLS data.^{20 21} The details are as follows: (1) weakness is assessed through a self-reported item: ‘have difficulty with lifting or carrying weights over 5 kg, like a heavy bag of groceries’; (2) slowness is determined by the participant’s difficulty in walking 100 metres or climbing several flights of stairs without resting; (3) exhaustion is noted if the participant answered ‘Most or all of the time’ or ‘Occasionally or a moderate amount of the time’ to the questions: ‘I felt everything I did was an effort’ or ‘I could not get going’; (4) low physical activity is defined as either the complete absence of physical activity or engaging in walking sessions that last <10 min each throughout the week and (5) weight loss is defined as an involuntary loss of 5 kg or more over the past year, or a current body mass index of 18.5 kg/m² or lower. Frailty is characterised by the presence of three or more of these criteria.

Predictors

The predictors are based on our previous systematic review and expert consultation. From the systematic review, we included the variables significantly predictive of frailty among older adults with diabetes. Furthermore, we consult with two diabetologists (LZ and XW) to finalise the selection of predictors. Online supplemental table 1 presents the researched domains, predictors, the measurement method and the planned handling of variables in the statistical analysis.22,24

Sample size

The sample size of the development dataset must be large enough to develop a prediction model equation that is reliable when applied to new individuals in the target population.²⁵ For the binary outcome prediction model, the sample size calculation is conducted as described in a previous study,²⁶ as follows: ${\displaystyle n=\mathrm{e}\mathrm{x}\mathrm{p}⟮\frac{-0.508+0.259\mathrm{I}\mathrm{n}(\mathrm{\Phi })+0.504\mathrm{I}\mathrm{n}\mathit{(}\mathit{P}\mathit{)}\mathit{-}\mathrm{I}\mathrm{n}\mathit{(}\mathit{M}\mathit{A}\mathit{P}\mathit{E}\mathit{)}}{0.544}⟯}$ Based on the above formula, the estimated minimum sample size required to ensure that prediction models will yield a low prediction error in estimating outcome probabilities for the target population (as measured by the mean absolute prediction error, MAPE) is calculated to be 375. Besides, the sample size for external validation should be sufficient to accurately assess model performance.²⁷ Therefore, we calculated the model external validation sample size according to the Riley et al²⁷ recommendations and the Stata code provided by Ensor.²⁸ Stata code of sample size calculations is: ‘pmvalsampsize, type(b) prevalence(.28) cstatistic(.88) lpbeta(1.55,1.85) oeciwidth(.30) csciwidth(.4) cstatciwidth(.1)’

As a result, at least 558 participants (153 events) are required for the external validation study to ensure calibration is properly evaluated.

Patient and public involvement

Patients and the public are not involved in the design, conduct, reporting or dissemination plans of this research, as the study is a secondary analysis of publicly available data from the CHARLS cohort.

Missing data

If more than 10% of the dataset consists of incomplete records, imputation procedures will be employed. If the data are missing at random or better, a multiple imputation approach will be used. For continuous variables, missing values will be addressed using the proximity point mean method.

Statistical analysis methods and model-building procedures

Predictors will be used as feature parameters for inclusion in the prediction model. Continuous variables will be standardised, while categorical variables will be handled using one-hot encoding methods. Employing recursive feature elimination, the study will identify the optimal subset to obtain the most favourable combination of features. Eight ML algorithms (Logistic Regression (LR), Support Vector Machine (SVM), Random Forest (RF), K-nearest Neighbour (KNN), Adaptive Boosting (AdaBoost), Extreme Gradient Boosting (XGBoost), Light Gradient Boosting Machine (LightGBM) and Decision Tree (ID3)) will be used to develop the prediction model for frailty in older adults with diabetes. We select these eight models to represent diverse methodological paradigms, each with distinct strengths in addressing data complexity. This comparative approach facilitates the identification of the most suitable model and is consistent with current best-practice guidelines for transparent and robust prediction modelling, as recommended by TRIPOD-AI statement. Model performance will be evaluated using receiver operating characteristic (ROC) curves, calibration plots and internal validation with leave-one-out cross validation. To select the optimal model, we will prioritise the area under the ROC curve (AUC) as the primary criterion for discrimination. If models demonstrate similar AUC values, we will further evaluate calibration using calibration plots and Brier scores. When discrimination and calibration are comparable, decision curve analysis will be used to guide selection based on clinical utility. This approach follows TRIPOD-AI recommendations to ensure transparency and clinical relevance in model development. Afterwards, the model will be evaluated using an external validation dataset derived from later waves of the CHARLS (eg, 2018 or 2020), which are not used in model development, to ensure temporal separation and enhance generalisability. The SHAP method will be employed to illustrate the relative importance of each variable, providing interpretable insights within the model. Finally, the optimal model will be deployed via an electronic risk calculator with SHAP-based visualisations to enhance clinical interpretability and support decision-making in routine practice.

Discussion

Currently, China has the highest number of older adults with diabetes in the world, totalling 35.5 million. This number is projected to increase to 78.1 million by 2045.¹ Early identification of individuals at high risk of frailty is important for delaying the onset, reducing complications and improving the prognosis of older adults with diabetes. To the best of our knowledge, this study aims to provide a thorough assessment of frailty risk in older adults with diabetes using the latest ML algorithms. Unlike other studies that have identified predictive variables through univariable logistic regression analysis, this study stands out by selecting model predictors based on meta-analysis and systematic review. Meta-analysis is an important research method and one of the strongest sources of evidence in evidence-based medicine.²⁹ This may help us identify more useful predictor variables. Furthermore, in the realm of ML, data are the essential component driving model training and performance improvement. The research data source is the CHARLS led by Peking University from 2011 to 2020, which is a high-quality database with a good representation of China’s middle-aged and older adults. Consequently, we believe our model holds significant potential for clinical application.

This study has some anticipated limitations. First, due to its retrospective nature, some data may be missing. Second, the number of available predictors may be limited, potentially leading to model underfitting. Despite these limitations, we expect this study to represent a valuable scientific contribution to the field of geriatric diabetes care.