Skip to main content
BMJ Open logoLink to BMJ Open
. 2023 May 18;13(5):e068465. doi: 10.1136/bmjopen-2022-068465

Development and validation of a novel nomogram of 1-year mortality in the elderly with hip fracture: a study of the MIMIC-III database

Qian Chen 1, Peng Hao 2, Chipiu Wong 1, Xiaoxin Zhong 2, Qing He 2,, Yantao Chen 1,
PMCID: PMC10201249  PMID: 37202145

Abstract

Objective

Hip fracture is a prevalent condition with a significant death rate among the elderly. We sought to develop a nomogram-based survival prediction model for older patients with hip fracture.

Design

A retrospective case–control study.

Setting

The data from Medical Information Mart for Intensive Care III (MIMIC-III V.1.4).

Participants

The clinical features of elderly patients with hip fracture, including basic information, comorbidities, severity score, laboratory tests and therapy, were filtered out based on the MIMIC-III V.1.4.

Methods and main outcome measures

All patients included in the study were from critical care and randomly divided into training and validation sets (7:3). On the basis of retrieved data, the least absolute shrinkage and selection operator (LASSO) regression and multiple logistic regression analysis were used to identify independent predictive variables of 1-year mortality, and then constructed a risk prediction nomogram. The predictive values of the nomogram model were evaluated by the concordance indexes (C-indexes), receiver operating characteristic curve, decision curve analysis (DCA) and calibration curve.

Results

A total of 341 elderly patients with hip fracture were included in this study; 121 cases died within 1 year. After LASSO regression and multiple logistic regression analysis, a novel nomogram contained the predictive variables of age, weight, the proportion of lymphocyte count, liver disease, malignant tumour and congestive heart failure. The constructed model proved satisfactory discrimination with C-indexes of 0.738 (95% CI 0.674 to 0.802) in the training set and 0.713 (95% CI 0.608 to 0.819) in the validation set. The calibration curve shows a good degree of fitting between the predicted and observed probabilities and the DCA confirms the model’s clinical practicability.

Conclusions

The novel prediction model provides personalised predictions for 1-year mortality in elderly patients with hip fractures. Compared with other hip fracture models, our nomogram is particularly suitable for predicting long-term mortality in critical patients.

Keywords: adult intensive & critical care, geriatric medicine, hip, risk factors, aged


STRENGTHS AND LIMITATIONS OF THIS STUDY.

  • We developed the first 1-year mortality prediction nomogram in elderly critical patients with hip fracture.

  • We used least absolute shrinkage and selection operator regression, as well as multivariable logistic regression, to screen for independent risk factors.

  • To evaluate the performance of the novel nomogram model in both the primary and validation cohorts, we included the area under the receiver operating characteristic curve, calibration curve and decision curve analysis.

  • Essentially, this is a single-centre retrospective study where data were collected from patients’ medical record databases. We were unable to control for data accuracy and bias.

Introduction

Hip fracture is a prevalent type of fracture in the elderly, and the incidence of hip fracture globally is predicted to approach 6 million per year by 2050 as the population ages.1 Hip fracture can cause high rate of disability and death, with reported rates of up to 50%2 and 12%–37% at 1 year3 4 and 47% at 2 years.5 Therefore, hip fracture has become a major public health issue, with rising healthcare expenses as a result. More than 3.5 times as many elderly patients with hip fracture die within 1 year than do elderly patients without hip fracture.6 The decline of physical function and underlying diseases in the elderly can exacerbate the risk of surgery, poor postoperative survival and quality of life,7 8 while conservative treatment increases the time spent in bed which may increase the risk of complications. Therefore, the severity of hip fracture itself and complications such as pulmonary infections, decubitus ulcers and embolism caused by prolonged bed rest can be fatal for the patients,9 10 so many patients may be admitted to the intensive care unit (ICU) for treatment.11

Many studies have analysed the risk factors associated with mortality or complications after hip fracture. Chang et al12 conducted a meta-analysis including 16 articles involving 25 349 patients, and concluded that the time to surgery, residential status (nursing home/home), cardiovascular disease, pulmonary disease and malignancy are preventable risk factors of mortality for patients with hip fracture. Another meta-analysis13 involving 18 articles suggested that senile dementia had a significant impact on the short-term and long-term mortality of hip fracture patients. Some studies have developed predictive models for postoperative pneumonia, blood transfusion or delirium for patients with hip fracture.14–16 It was reported that the American society of Aneshesiologists (ASA) physical status classification system, the Visual Analogue Scale for Risk scale and the Charlson Complication Index (CCI) were associated with complications and mortality after hip fracture.17 18 However, those models have some shortcomings, such as unsatisfactory predictive efficacy and limited application.

There is no predictive model constructed for 1-year mortality in elderly critical patients with hip fracture. Nomogram model can provide a visualisation tool for clinical prognostic studies to individually calculate the likelihood of clinical events occurring in each patient.19 Based on the Medical Information Mart for Intensive Care III (MIMIC-III, V.1.4), a large clinical database, this study systematically analysed the correlation between clinical baseline characteristics and 1-year mortality of critical patients with hip fracture and construct and validated a predictive nomogram model. The predictive model can assist clinicians in determining the prognosis of hip fracture patients who require ICU admission and has the potential to aid in identifying those at high risk of mortality.

Materials and methods

Sources of data

The MIMIC-III is an open critical care medicine database that was collaboratively released by the Computational Physiology Laboratory at the Massachusetts Institute of Technology (MIT), Beth Israel Deacon Medical Center (BIDMC) and Philips Medical with the support of the National Institutes of Health. From June 2001 to October 2012, the MIMIC- III database gathered hospitalisation information for more than 50 000 patients hospitalised to the ICU at BIDMC in the USA.

Patient and public involvement

Patients and/or the public were not directly involved in this study.

Data collection and definitions

Data extraction was performed using PostgreSQL V.9.6 software. The following data were extracted or calculated: basic information, vital signs, comorbidities, laboratory parameters, treatment and the severity score. Essential information included patient identification number, gender, age, ethnicity, weight, the time of admission and discharge of hospital and ICU, the day of death, and the diagnosis (category of fracture). The vital signs included systolic blood pressure, diastolic blood pressure, mean arterial pressure, SPO2, heart rate, respiratory rate (RR), temperature and blood glucose. The comorbidities included hypertension, diabetes mellitus, congestive heart failure, shock, chronic obstructive pulmonary disease, renal failure and malignant tumour. The laboratory parameters included white cell count (WCC), the proportion of neutrophil and lymphocyte count, haemoglobin, haematocrit, platelet, blood urea nitrogen (BUN), creatinine, international normalised ratio (INR), prothrombin time (PT), partial thromboplastin time (PTT). The severity score, simplified acute physiology score II (SAPS II), sequential organ failure as severity (SOFA) and acute physiology score III (APS III) were calculated. The treatment included the use of mechanical ventilation, vasopressor and the surgery for hip fracture. Indicators with missing values greater than 20% are removed, such as height, lactate, calcium and albumin. All data were extracted from the critical value of the patient during the first 24 hours after admission to ICU. Inclusion criteria were as follows: (1) the patients were diagnosed with hip fracture, including acetabular fracture, femoral neck fracture, intertrochanteric fracture and subtrochanteric fracture and (2) first admission to ICU. Exclusion criteria were as follows: (1) age <60 years old and (2) pathological fracture. The primary outcomes were all-cause 1-year mortality which was determined by time of admission and time of death recorded in the database. The assessors were blind to predictor results.

Nomogram construction

The retrieved patients were then randomly split into two groups: the training set and the validation set, with a 7:3 ratio between the two. All data extracted from the database were compared between survival and death groups and between training and validation sets. Predictors of 1-year mortality were screened out by means of the least absolute shrinkage and selection operator (LASSO) regression. After the cross-validation results, screening variables were retained when lambda is the optimal value. The multiple logistic regression model with the stepwise regression was established using these variables. The Hosmer-Lemeshow (HL) test was used to determine the goodness of fit of the multiple logistic regression model. Finally, a nomogram model was developed with the statistically significant parameters in the result of multiple logistic regression.

Nomogram validation and performance evaluation

The receiver operating characteristic (ROC) curve was generated to evaluate the recognition capability of the nomogram and ICU clinical score SAPS II, SOFA and APS III on the basis of the areas under the ROC curve (AUC). It was determined how well the nomogram model worked by computing its sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV). The calibration curve was generated to evaluate the consistency between the actual and anticipated incidence; at the same time, the corrected concordance index (C-index) was calculated via 1000 bootstrap resamples. Finally, we used decision curve analysis (DCA) to assess the clinical utility of the nomogram model as well as the ICU clinical scores SAPS II, SOFA and APS III. All patients included in this study were scored based on the nomogram model and divided into low-risk, intermediate-risk and high-risk groups according to nearlyone-third of the population.

Statistical analysis

For continuous variables, the normality test was performed first by the Shapiro-Wilk test. Comparisons of continuous variables were made using either the t-test or the Mann-Whitney U test, depending on the circumstances, while comparisons of categorical variables were made using χ2 or Fisher’s exact tests. Continuous variables were present as median (IQRs), while categorical variables were presented as frequency and percentage. The ‘MICE’ R package was used to implement the neighbour interpolation approach, which was then used to fill in any missing data.20 The significance level was set at 0.05 for all tests carried out using R (V.4.0.2; https://www.r-project.org).

Results

Subject characteristics

The flow chart of the study is shown in figure 1. An average age of 81.7 years was found among the 341 patients who had been diagnosed with hip fracture in our research, which had 1-year death rate of 35.5%. There are comparisons in characteristics between the death group and the survival group shown in table 1. Compared with the survival group, the patients in the death group were older, lighter in weight, faster in RR, lower in lymphocyte count ratio and longer in hospital and ICU stay. Meanwhile, higher prevalence of congestive heart failure and liver disease was detected in the death group. Additionally, the values of the indicators linked to coagulation function INR, PT and PTT were considerably higher, which suggests that there was an impairment in coagulation function in the death group. Moreover, the ICU clinical scores SOFA, SAPS II and APS III were significantly higher, indicating that the death group was more critical. Based on a 7:3 random allocation ratio, the training and test sets were 238 and 103 patients, with 89 and 32 deaths, respectively. There were no significant differences found between the training set and the validation set, and online supplemental table 1 presents a comparison of the fundamental properties of both sets.

Figure 1.

Figure 1

The flow chart of the study. AUC, area under the curve; HL, Hosmer-Lemeshow; ICU, intensive care unit; LASSO, least absolute shrinkage and selection operator; MIMIC-III, Medical Information Mart for Intensive Care III; NPV, negative predictive value; PPV, positive predictive value; ROC, receiver operating characteristic.

Table 1.

The essential characteristics of 1-year death and survival groups in hip fracture

Variables All (n=341) Survive (n=220) Death (n=121) P value
Basic hospitalisation information
 Age, years 81.7 (73.3, 87.2) 79.7 (71.8, 86.1) 84.4 (78.3, 88.7) <0.001
 Sex, n (%) 0.649
  Female 186 (54.5) 118 (53.6) 68 (56.2)
  Male 155 (45.5) 102 (46.4) 53 (43.8)
 Weight, kg 68.0 (58.0, 82.0) 71.00 (60.0, 87.0) 64.00 (55.0, 75.0) <0.001
 Ethnicity, n (%) 0.67
  Black 14 (4.1) 9 (4.1) 5 (4.1)
  White 289 (84.8) 184 (83.6) 105 (86.8)
  Other 38 (11.1) 27 (12.3) 11 (9.1)
 ICU stay time, days 2.6 (1.3, 4.7) 2.3 (1.3, 4.5) 2.9 (1.3, 5.2) 0.414
 Hospital stay time, days 8.8 (5.9, 12.8) 8.2 (5.8, 12.7) 9.6 (6.2, 13.8) 0.457
 Category of fracture, n (%) 0.035
  Intertrochanteric fracture 126 (37.0) 74 (33.6) 52 (43.0)
  Femoral neck fracture 106 (31.1) 66 (30.0) 40 (33.1)
  Subtrochanteric fracture 31 (9.1) 19 (8.6) 12 (9.9)
  Acetabular fracture 78 (22.9) 61 (27.7) 17 (14.0)
Severity of illness
 APSIII 46.0 (36.0, 58.0) 43.0 (34.0, 55.0) 50.0 (40.0, 62.0) 0.002
 SOFA 4.0 (2.0, 6.0) 4.0 (2.0, 6.0) 5.0 (3.0, 7.0) 0.002
 SAPSII 42.0 (33.0, 49.0) 40.0 (30.8, 46.0) 46.0 (36.0, 54.0) <0.001
Vital signs
 Heart rate, beats/min 86.04 (75.52, 96.02) 86.32 (75.46, 95.91) 85.72 (75.73, 96.33) 0.686
 SBP, mm Hg 115.9 (106.9, 127.1) 116.5 (107.2, 127.4) 113.1 (103.8, 126.4) 0.137
 DBP, mm Hg 55.81 (48.72, 62.38) 56.13 (49.64, 62.67) 55.57 (47.79, 61.61) 0.313
 MAP, mm Hg 72.54 (66.85, 80.85) 72.52 (67.57, 80.50) 72.90 (65.70, 82.07) 0.864
 Resprate rate, breaths/min 18.6 (16.2, 21.1) 18.2 (15.8, 20.4) 19.3 (17.3, 22.0) 0.002
 Temperature, ℃ 37.4 (37.0, 37.9) 37.4 (37.1, 37.9) 37.4 (36.9, 38.0) 0.408
 Glu, mEq/L1 143.0 (121.0, 167.1) 145.4 (123.0, 168.7) 137.8 (117.2, 162.5) 0.161
 Spo2, % 92.0 (87.0, 94.0) 92.0 (88.0, 94.0) 91.00 (86.0, 94.0) 0.134
Comorbidities, n (%)
 Congestive heart failure 0.012
  No 208 (61.0) 145 (65.9) 63 (52.1)
  Yes 133 (39.0) 75 (34.1) 58 (47.9)
 Shock 0.624
  No 332 (97.4) 213 (96.8) 119 (98.3)
  Yes 9 (2.6) 7 (3.2) 2 (1.7)
 Hypertension 0.371
  No 144 (42.2) 89 (40.5) 55 (45.5)
  Yes 197 (57.8) 131 (59.5) 66 (54.5)
 COPD 0.267
  No 249 (73.0) 165 (75.0) 84 (69.4)
  Yes 92 (27.0) 55 (25.0) 37 (30.6)
 Diabetes mellitus 0.898
  No 241 (70.7) 156 (70.9) 85 (70.2)
  Yes 100 (29.3) 64 (29.1) 36 (29.8)
 Renal failure 0.203
  No 272 (79.8) 180 (81.8) 92 (76.0)
  Yes 69 (20.2) 40 (18.2) 29 (24.0)
 Liver disease 0.046
  No 329 (96.5) 216 (98.2) 113 (93.4)
  Yes 12 (3.5) 4 (1.8) 8 (6.6)
 Malignant tumour 0.141
  No 319 (93.5) 209 (95.0) 110 (90.9)
  Yes 22 (6.5) 11 (5.0) 11 (9.1)
Laboratory tests
 INR 1.3 (1.1, 1.8) 1.2 (1.1, 1.6) 1.4 (1.2, 1.9) 0.001
 PTT, seconds 32.2 (27.2, 40.0) 31.0 (26.6, 37.6) 34.1 (28.4, 49.0) 0.001
 PT, seconds 14.6 (13.3, 17.6) 14.2 (13.1, 16.8) 15.4 (13.8, 18.4) 0.001
 Haemoglobin, g/dL 9.4 (8.3, 10.5) 9.4 (8.2, 10.4) 9.5 (8.8, 10.5) 0.358
 Haematocrit, % 33.0 (30.1, 37.2) 33.1 (30.1, 37.3) 32.7 (30.1, 36.5) 0.647
 Creatinine, mg/dL 1.1 (0.8, 1.6) 1.1 (0.8, 1.5) 1.2 (0.9, 1.7) 0.272
 BUN, mg/dL 25.0 (18.0, 38.0) 25.0 (18.0, 33.3) 28.0 (19.0, 44.0) 0.012
 WCC, 109/L 12.8 (9.9, 16.3) 12.1 (9.7, 15.7) 13.7 (10.5, 17.5) 0.135
 Neutrophil, % 84.5 (79.0, 89.3) 84.1 (77.0, 89.2) 85.2 (80.8, 89.5) 0.194
 Lymphocyte,% 8.9 (6.0, 14.0) 9.8 (6.35, 15.0) 8.0 (5.6, 11.3) 0.01
 Platelet count, 109/L 166.0 (110.0, 207.0) 160.5 (107.8, 200.0) 173.0 (120.0, 222.0) 0.081
Treatment, n (%)
 Ventilation 0.299
  No 182 (53.4) 122 (55.5) 60 (49.6)
  Yes 159 (46.6) 98 (44.5) 61 (50.4)
 Vasopressor use 0.868
  No 240 (70.4) 158 (71.8) 82 (67.8)
  Yes 101 (29.6) 62 (28.2) 39 (32.2)
 Surgery for fracture 0.868
  No 66 (19.4) 42 (19.1) 24 (19.8)
  Yes 275 (80.6) 178 (80.9) 97 (80.2)

APSIII, Acute Physiology Score III; BUN, blood urea nitrogen; COPD, chronic obstructive pulmonary disease; CRRT, continuous renal replacement therapy; DBP, diastolic blood pressure; INR, international normalised ratio; MAP, mean arterial pressure; PT, prothrombin time; PTT, partial thromboplastin time; SAPSII, Simplified Acute Physiology Score II; SBP, systolic blood pressure; SOFA, Sequential Organ Failure Assessment; WCC, white cell count.

Supplementary data

bmjopen-2022-068465supp001.pdf (86.8KB, pdf)

Characteristics selection and development of a nomogram

The LASSO regression was applied to the pool of 34 candidates (all parameters in table 1 except hospital and ICU stay and severity of illness), and 8 variables were ultimately chosen (a ratio of 4.3:1) (figure 2). After multiple logistic analysis of the eight variables, the final six variables with statistical significance were: age (OR 1.047; 95% CI 1.008 to 1.090), weight (OR 0.981; 95% CI 0.964 to 0.997), the proportion of lymphocyte count (OR 0.962; 95% CI 0.922 to 0.997), liver disease (OR 18.529; 95% CI 3.053 to 358.170), congestive heart failure (OR 2.214; 95% CI 1.224 to 4.050), malignant tumour (OR 4.264; 95% CI 1.382 to 14.207) (table 2). We built a nomogram to predict 1-year mortality for older patients who had hip fracture using a multivariable logistic regression model that served as our basis (figure 3).

Figure 2.

Figure 2

Lasso regression screening variables: screening retention variables when lambda is the optimal value. (A) tuning parameter (λ) selection using LASSO penalised logistic regression with 10-fold cross-validation. (B) LASSO coefficient profiles of the radiomic features. LASSO, least absolute shrinkage and selection operator.

Table 2.

Multivariable regression model based on the LASSO regression results

Variables Multivariable logistics model base on LASSO regresson result Multivariable logistics model base on stepwise regression result
OR 95% CI P value OR 95% CI P value
Age 1.041 1.001 to 1.084 0.047 1.047 1.008 to 1.090 0.021
Weight 0.981 0.964 to 0.998 0.037 0.981 0.964 to 0.997 0.027
Lymphocyte 0.963 0.922 to 0.998 0.059 0.962 0.922 to 0.997 0.054
Liver disease 17.135 2.831 to 330.659 0.01 18.529 3.053 to 358.170 0.008
Congestive heart failure 1.877 1.008 to 3.52 0.048 2.214 1.224 to 4.050 0.009
Malignant tumour 4.228 1.345 to 14.341 0.015 4.264 1.382 to 14.207 0.013
BUN 1.012 0.996 to 1.028 0.134
Diagnosis 0.825 0.627 to 1.079 0.165

BUN, blood urea nitrogen; LASSO, least absolute shrinkage and selection operator.

Figure 3.

Figure 3

Nomogram predicting 1-year mortality of the elderly patients wtih hip fracture.

Apparent performance of the nomogram

The AUC value of the nomogram was 0.738 (95% CI 0.674 to 0.802) for the training set. The adjusted C-index was 0.717 after 1000 bootstrap resamples. The AUC also satisfactorily reached 0.713 (95% CI 0.608 to 0.819) in the validation dataset. The AUC for the scoring systems APS III, SOFA and SAPS II for all data were 0.603, 0.599 and 0.646, respectively (online supplemental figure 1), suggesting that the constructed model showed better predictive performance than clinical scores commonly used in ICU. On the basis of the maximum of the Youden index, the ideal cut-off values for the nomogram predicted probability were established at 65.2% in the training set and 40.6% in the validation set correspondingly. The sensitivity, specificity, PPV and NPV of the ROC for the nomogram were 78.7%, 60.4%, 54.3%, 82.6% in the training dataset, and 59.4%, 78.9%, 55.9%, 81.1% in the validation set showing in online supplemental table 2. Both cohorts displayed a satisfactory degree of fitting with the nomogram, as evidenced by the calibration curve (online supplemental figure 2). In addition, a satisfactory consistency was found between the predicted and actual values in both the training set (χ2=6.34, p=0.609) and the validation set(χ2=4.09, p=0.849) via the HL test for the multiple logistic regression model.

Supplementary data

bmjopen-2022-068465supp002.pdf (127KB, pdf)

Supplementary data

bmjopen-2022-068465supp003.pdf (60KB, pdf)

Supplementary data

bmjopen-2022-068465supp004.pdf (95.1KB, pdf)

Clinical practice

All patients included in this study were divided into the low risk (108/341, 31.67%; score: 0–190), the intermidiate risk (120/341, 35.19%; score of 190.1–220), the high risk (60/113 33.14%; score: >220.1). The mortality rates among the three groups were 16.7% (18/108), 35.8% (43/120) and 53.1% (60/113), respectively, with significant statistical differences (p<0.001), which confirmed that the model could distinguish patients with high risk of death. The DCA plot was generated to exhibit great positive net benefits of this nomogram compared with clinical severity scores. The horizontal black line represents the assumption that no patients receive the intervention being evaluated, so the net benefit is 0. The oblique dotted line represents the assumption that all patients receive the intervention. The green and pink curves show the clinical benefit of the model’s predictions in the training and test sets, respectively. The blue, purple and red curves represent the clinical benefit for the three critical scores, respectively. Clinical intervention guided by the nomogram provided a better clinical application than the clinical scoring system APS III, SOFA and SAPS II when the threshold probability was within 0.1 and 1.0 and within 0.1 and 0.5 in the training and validation set (online supplemental figure 3).

Supplementary data

bmjopen-2022-068465supp005.pdf (86.8KB, pdf)

Discussion

In this study, we retrospectively analysed 341 elderly patients with hip fracture based on the MIMIC-III (V.1.4) database and screened independent risk factors for 1-year mortality by LASSO regression and multiple logistic regression, from which a best-fit nomogram model was constructed with a total of 6 clinical variables, that is, the age, weight, the proportion of lymphocyte count, combined with liver disease, congestive heart failure, malignant tumour. Patients in this study had a death rate of 35.4% at 1 year, which was much higher than some studies, partly because all of the data in this study was taken from critically ill patients in the ICU. Compared with the APS III, SOFA and SAPS II scales commonly used in critical care medicine, the constructed model has greater accuracy and net clinical benefit for the 1-year death rate in senior patients with hip fracture who have been admitted to the ICU for treatment.

Similar to the findings of other studies,21–23 age was shown to be a significant predictor of 1-year death in the elderly with hip fracture, with a 4.1% increase in mortality each year of increasing age in this study. As age increases, patients’ physiological function and immune capacity decrease, the number of comorbid medical diseases increases, and stress capacity to trauma, anaesthesia and surgery are poor, leading to a high incidence of complications, a high disability rate and a high mortality rate after hip fracture.

It has been suggested that weight loss is a significant risk factor for hip fracture in older adults.24 A meta-analysis including eight prospective studies concluded that weight change deeply impacts on the prognosis of patients with hip fracture; body weight loss and gain might be a risk and protective factor for hip fracture, respectively.25 Another meta-analysis including 11 independent studies concluded that obese patients following hip fracture surgery had lower mortality both in the long term and short term.26 This inverse relationship between body weight and mortality is known as the obesity paradox.27 Similar conclusions were reached in this study, but height data were severely missing, and only weight was included in the analysis; the lower the weight, the worse the prognosis of the patient. Several studies have explained the physiopathology of the obesity paradox in terms of the influence of weight on mortality following hip fracture surgery. After trauma or surgery, patients may suffer from a series of physiopathological responses caused by inflammatory reactions if that is excessive can lead to systemic inflammatory response syndrome (SIRS) may induce systemic multiple organ failure and ultimately death. Besides, patients could reach a high catabolic state for up to 3 months, while a lot of adipose tissue in obese patients serves as an energy reserve to provides the substances needed for increased metabolism after hip fracture. Therefore, obese patients have a greater capacity to tolerate the stress response after hip fracture.28 29 In addition, in the elderly, being underweight is probably related to poor nutrition and osteoporosis30 which increases the risk of bone fracture. Elderly patients with frailty and malnutrition are less capable of functional rehabilitation and fracture healing following hip fracture surgery.31

A meta-analysis showed that total lymphocyte count (TLC) was shown to be an indicator of higher mortality in individuals who had hip fracture surgery.32 Atlas et al concluded that postoperative 5th day neutrophil-lymphocyte ratios (NLR) and the rate of increase of that was higher among patients who were admitted to the ICU and among those who died.33 Blood cells are mainly composed of neutrophils/lymphocytes, and when lymphocyte count is low, neutrophil count is relatively high, and NLR is increased, indirectly reflecting the prognostic impact of low lymphocyte count. Another meta-analysis incorporating 19 studies analysed came to a similar conclusion that higher preoperative and postoperative NLR were both related to higher long-term mortality after hip fracture in the elderly population.34 Delayed wound healing following hip fracture was associated with lower serum TLC, which has a significant predictive value.35 Systemic inflammation is known to be associated with prognosis after hip fracture surgery.36 TLC has been used as a marker of immune status, and preoperative lymphopenia is considered an important risk factor for postoperative sepsis and mortality.37 some researchers have reported the pathophysiology of similar systemic acute inflammatory markers was associated with prognosis after hip fracture, such as tumour necrosis factor-α (TNF-α) and interleukin-6 (IL-6).38 39 The conclusions above are basically consistent with our research.

Liver disease was the most severe risk factor for mortality in patients with hip fracture in this study. Several studies40–45 have shown that patients with hip fracture combined with liver disease had a 2–5 fold increased risk of death in the short term or long term compared with patients without liver disease. Patients who have liver illness have a higher risk for serious complications that might jeopardise their lives, including venous thromboembolism, hypoglycaemia, hypercoagulability, infection and malnutrition.46 Patients with impaired liver function are prone to osteoporosis due to osteoclast-mediated bone resorption and bilirubin-related inhibition of osteoblast proliferation,47 and impaired bone metabolism may affect hip fracture healing, leading to delayed mobility and deteriorating health status to the point of affecting long-term mortality.48

Congestive heart failure and malignancy are known to be independent predictors of mortality in many diseases, and hip fracture is no exception.49–52 Heart disease and malignancy make patients less resistant to trauma and surgery and less able to heal fracture.

Over the years, several risk assessment tools have been developed to predict mortality in patients with hip fracture. Karres et al53 evaluated six risk prediction models for 30-day mortality following hip fracture surgery, including the Nottingham Hip Fracture Score (NHFS), Orthopaedic Physiologic and Operative Severity Score for the enUmeration of Mortality and Morbidity(O-POSSUM), Estimation of Physiologic Ability and Surgical Stress, and the CCI and two other models that have been reported in the papers. They found that the NHFS was the most promising models with reasonable discrimination and extensive validation for predicting 30-day mortality in hip fracture surgery. Nia et al54 compared four risk prediction scores for 30-day and 180-day mortality in elderly proximal hip fracture patients, including the POSSUM, the Portsmouth-POSSUM (P-POSSUM), the CCI and National Surgical Quality Improvement Programme (NSQIP) Risk Score. They found that ACS-NSQIP had the highest predictive value for both 30-day and 180-day mortality in elderly proximal hip fracture patients. Ramanathan et al55 evaluated the POSSUM scoring system in patients with fractured neck of femur and found that it was a useful tool for predicting morbidity and mortality in this population. Wanjiang et al56 conducted a systematic review and meta-analysis on the application of POSSUM and P-POSSUM scores in the risk assessment of elderly hip fracture surgery. They found that the POSSUM and P-POSSUM scores had moderate predictive value for morbidity and mortality in elderly hip fracture surgery, but further studies were needed to validate their performance. While these models have demonstrated good predictive performance in some studies, their generalisability to different patient populations and settings is uncertain. In comparison with these existing models, our nomogram predictive model has advantages in predicting long-term mortality (1 year), and includes novel predictors of lymphocyte ratio, and targeted for the critical old patients with hip fracture, with good significance in distinguishing, calibrating and clinical validity. In addition, it is worth noting that the variables included in our nomogram model were selected based on their clinical relevance and statistical significance in LASSO regression and multivariable logistic regression. This differs from the risk prediction models evaluated in the reference articles. Otherwise, the six variables that were ultimately included in the model are readily obtainable during routine clinical treatment, and the nomogram is particularly advantageous as a visual aid for clinical application. As such, it may provide a valuable contribution to the field.

Similar to other studies based on the MIMIC-III database, the study also has certain limitations: (1) all of the patients who participated in this research came from the MIMIC-III database, so all the data collected in this single-centre retrospective study are not as precise as the dataset analysis collected in prospective cohort study; (2) some data were missing or not included in the database, such as lactate, albumin, height, ASA score and time of surgery, which can not be included in this analysis, limiting the validity of the comparison; (3) this study failed to analyse the dynamic changes of various laboratory indicators and scores which more directly reflect the prognosis of patients and (4) the model constructed by us is only verified internally, and further verification is needed in another independent population in the future. Overall, while our study adds to the growing body of research on mortality prediction in hip fracture patients, further validation of our nomogram model in external patient populations is necessary to assess its generalisability and clinical utility. Additionally, future studies could explore the potential value of combining our model with other risk prediction tools to enhance their accuracy and reliability.

Conclusions

According to the findings of our research, factors such as liver illness, congestive heart failure and malignant tumour, as well as age, weight and the percentage of lymphocyte count, are all significant predictors of death in older populations with hip fracture after 1 year. Based on these risk factors, we established and validated a nomogram model. The clinical doctor can screen out high-risk patients and make making optimal medical decisions for treatment and rehabilitation using the model.

Supplementary Material

Reviewer comments
Author's manuscript

Acknowledgments

Thanks to Prof. Lv Jun from the Clinical Research Department of Jinan University for his guidance on data extraction and related software use.

Footnotes

Contributors: QC collected data, analysed it and wrote the manuscript. PH, CW were involved in the study’s conception, design and coordination, as well as helping to draft the paper.XZ assisted in data collection and paper revision.QH and YC were in charge of the entire project, reviewing the article, planning the study, and supervising it. All of the authors contributed to the paper and gave their approval to the final version.YT had ultimate oversight for the design andconduct and writing of this manuscript.

Funding: This work was supported by Clinical Medicine Research Fund of Guangdong Province (grant number: 20212T14).

Competing interests: None declared.

Patient and public involvement: Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

Provenance and peer review: Not commissioned; externally peer reviewed.

Supplemental material: This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Data availability statement

Data are available on reasonable request. The datasets generated and analysed during the current study are not publicly available due the fact that they constitute an excerpt of research in progress but are available from the corresponding author on reasonable request.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

As all patients' records are anonymised, individual permission was not required for this retrospective analysis, which was authorised by the BIDMC and the Institutional Review Boards of MIT following completion of the CITI (Record ID#40362987).

References

  • 1.Cooper C, Cole ZA, Holroyd CR, et al. Secular trends in the incidence of hip and other osteoporotic fractures. Osteoporos Int 2011;22:1277–88. 10.1007/s00198-011-1601-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Leung F, Blauth M, Bavonratanavech S. Surgery for fragility hip fracture-streamlining the process. Osteoporos Int 2010;21:S519–21. 10.1007/s00198-010-1402-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Hong SE, Kim TY, Yoo JH, et al. Acute kidney injury can predict in-hospital and long-term mortality in elderly patients undergoing hip fracture surgery. PLoS One 2017;12:e0176259. 10.1371/journal.pone.0176259 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Lunde A, Tell GS, Pedersen AB, et al. The role of comorbidity in mortality after hip fracture: a nationwide Norwegian study of 38,126 women with hip fracture matched to a general-population comparison cohort. Am J Epidemiol 2019;188:398–407. 10.1093/aje/kwy251 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Ireland AW, Kelly PJ, Cumming RG. Risk factor profiles for early and delayed mortality after hip fracture: analyses of linked Australian department of Veterans’ Affairs databases. Injury 2015;46:1028–35. 10.1016/j.injury.2015.03.006 [DOI] [PubMed] [Google Scholar]
  • 6.Lystad RP, Cameron CM, Mitchell RJ. Mortality risk among older Australians hospitalised with hip fracture: a population-based matched cohort study. Arch Osteoporos 2017;12:67. 10.1007/s11657-017-0359-7 [DOI] [PubMed] [Google Scholar]
  • 7.Rocos B, Whitehouse MR, Kelly MB. Resuscitation in hip fractures: a systematic review. BMJ Open 2017;7:e015906. 10.1136/bmjopen-2017-015906 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Tarazona-Santabalbina FJ, Ojeda-Thies C, Figueroa Rodríguez J, et al. Orthogeriatric management: improvements in outcomes during hospital admission due to hip fracture. Int J Environ Res Public Health 2021;18:3049. 10.3390/ijerph18063049 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Berggren M, Karlsson Å, Lindelöf N, et al. Effects of geriatric interdisciplinary home rehabilitation on complications and readmissions after hip fracture: a randomized controlled trial. Clin Rehabil 2019;33:64–73. 10.1177/0269215518791003 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Karlsson Å, Lindelöf N, Olofsson B, et al. Effects of geriatric interdisciplinary home rehabilitation on independence in activities of daily living in older people with hip fracture: a randomized controlled trial. Arch Phys Med Rehabil 2020;101:571–8. 10.1016/j.apmr.2019.12.007 [DOI] [PubMed] [Google Scholar]
  • 11.Hasan O, Mazhar L, Rabbani U, et al. Does early surgery prevent postoperative ICU admission after surgery for the fracture of the hip. Nested case control study of 911 patients. Ann Med Surg (Lond) 2021;61:35–40. 10.1016/j.amsu.2020.12.017 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Chang W, Lv H, Feng C, et al. Preventable risk factors of mortality after hip fracture surgery: systematic review and meta-analysis. Int J Surg 2018;52:320–8. 10.1016/j.ijsu.2018.02.061 [DOI] [PubMed] [Google Scholar]
  • 13.Bai J, Zhang P, Liang X, et al. Association between dementia and mortality in the elderly patients undergoing hip fracture surgery: a meta-analysis. J Orthop Surg Res 2018;13:298. 10.1186/s13018-018-0988-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Xiang G, Dong X, Xu T, et al. A nomogram for prediction of postoperative pneumonia risk in elderly hip fracture patients. Risk Manag Healthc Policy 2020;13:1603–11. 10.2147/RMHP.S270326 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Zhang X, Tong D-K, Ji F, et al. Predictive nomogram for postoperative delirium in elderly patients with a hip fracture. Injury 2019;50:392–7. 10.1016/j.injury.2018.10.034 [DOI] [PubMed] [Google Scholar]
  • 16.Bian FC, Cheng XK, An YS. Preoperative risk factors for postoperative blood transfusion after hip fracture surgery: establishment of a nomogram. J Orthop Surg Res 2021;16:406. 10.1186/s13018-021-02557-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Burgos E, Gómez-Arnau JI, Díez R, et al. Predictive value of six risk scores for outcome after surgical repair of hip fracture in elderly patients. Acta Anaesthesiol Scand 2008;52:125–31. 10.1111/j.1399-6576.2007.01473.x [DOI] [PubMed] [Google Scholar]
  • 18.Kirkland LL, Kashiwagi DT, Burton MC, et al. The Charlson comorbidity index score as a predictor of 30-day mortality after hip fracture surgery. Am J Med Qual 2011;26:461–7. 10.1177/1062860611402188 [DOI] [PubMed] [Google Scholar]
  • 19.Zhou Z-R, Wang W-W, Li Y, et al. In-depth mining of clinical data: the construction of clinical prediction model with R. Ann Transl Med 2019;7:796. 10.21037/atm.2019.08.63 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Beretta L, Santaniello A. Nearest neighbor imputation algorithms: a critical evaluation. BMC Med Inform Decis Mak 2016;16 Suppl 3:74. 10.1186/s12911-016-0318-z [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Kawai M, Tanji A, Nishijima T, et al. Association between time to surgery and 90-day mortality after hip fracture: a retrospective cohort study of 1734 cases. J Orthop Sci 2018;23:987–91. 10.1016/j.jos.2018.07.016 [DOI] [PubMed] [Google Scholar]
  • 22.Beaupre LA, Khong H, Smith C, et al. The impact of time to surgery after hip fracture on mortality at 30- and 90-days: does a single benchmark apply to all? Injury 2019;50:950–5. 10.1016/j.injury.2019.03.031 [DOI] [PubMed] [Google Scholar]
  • 23.Lizaur-Utrilla A, Gonzalez-Navarro B, Vizcaya-Moreno MF, et al. Reasons for delaying surgery following hip fractures and its impact on one year mortality. Int Orthop 2019;43:441–8. 10.1007/s00264-018-3936-5 [DOI] [PubMed] [Google Scholar]
  • 24.Dai Z, Ang LW, Yuan JM, et al. Association between change in body weight after midlife and risk of hip fracture-the Singapore Chinese Health study. Osteoporos Int 2015;26:1939–47. 10.1007/s00198-015-3099-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Lv Q-B, Fu X, Jin H-M, et al. The relationship between weight change and risk of hip fracture: meta-analysis of prospective studies. Sci Rep 2015;5:16030. 10.1038/srep16030 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Yang T-I, Chen Y-H, Chiang M-H, et al. Inverse relation of body weight with short-term and long-term mortality following hip fracture surgery: a meta-analysis. J Orthop Surg Res 2022;17:249. 10.1186/s13018-022-03131-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Modig K, Erdefelt A, Mellner C, et al. "Obesity paradox'' holds true for patients with hip fracture. The Journal of Bone and Joint Surgery 2019;101:888–95. 10.2106/JBJS.18.01249 [DOI] [PubMed] [Google Scholar]
  • 28.Hébuterne X, Bermon S, Schneider SM. Ageing and muscle: the effects of malnutrition, re-nutrition, and physical exercise. Curr Opin Clin Nutr Metab Care 2001;4:295–300. 10.1097/00075197-200107000-00009 [DOI] [PubMed] [Google Scholar]
  • 29.Hedström M, Ljungqvist O, Cederholm T. Metabolism and catabolism in hip fracture patients: nutritional and anabolic intervention -- a review. Acta Orthop 2006;77:741–7. 10.1080/17453670610012926 [DOI] [PubMed] [Google Scholar]
  • 30.Coin A, Sergi G, Benincà P, et al. Bone mineral density and body composition in underweight and normal elderly subjects. Osteoporos Int 2000;11:1043–50. 10.1007/s001980070026 [DOI] [PubMed] [Google Scholar]
  • 31.Meesters DM, Wijnands KAP, Brink PRG, et al. Malnutrition and fracture healing: are specific deficiencies in amino acids important in nonunion development? Nutrients 2018;10:1597. 10.3390/nu10111597 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Li S, Zhang J, Zheng H, et al. Prognostic role of serum albumin, total lymphocyte count, and mini nutritional assessment on outcomes after geriatric hip fracture surgery: a meta-analysis and systematic review. J Arthroplasty 2019;34:1287–96. 10.1016/j.arth.2019.02.003 [DOI] [PubMed] [Google Scholar]
  • 33.Atlas A, Duran E, Pehlivan B, et al. The effect of increased neutrophil lymphocyte ratio on mortality in patients operated on due to hip fracture. Cureus 2020;12:e6543. 10.7759/cureus.6543 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Chen Y-H, Chou C-H, Su H-H, et al. Correlation between neutrophil-to-lymphocyte ratio and postoperative mortality in elderly patients with hip fracture: a meta-analysis. J Orthop Surg Res 2021;16:681. 10.1186/s13018-021-02831-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Guo JJ, Yang H, Qian H, et al. The effects of different nutritional measurements on delayed wound healing after hip fracture in the elderly. J Surg Res 2010;159:503–8. 10.1016/j.jss.2008.09.018 [DOI] [PubMed] [Google Scholar]
  • 36.Norring-Agerskov D, Bathum L, Pedersen OB, et al. Biochemical markers of inflammation are associated with increased mortality in hip fracture patients: the Bispebjerg hip fracture Biobank. Aging Clin Exp Res 2019;31:1727–34. 10.1007/s40520-019-01140-7 [DOI] [PubMed] [Google Scholar]
  • 37.Koval KJ, Maurer SG, Su ET, et al. The effects of nutritional status on outcome after hip fracture. J Orthop Trauma 1999;13:164–9. 10.1097/00005131-199903000-00003 [DOI] [PubMed] [Google Scholar]
  • 38.Forget P, Moreau N, Engel H, et al. The neutrophil-to-lymphocyte ratio (NLR) after surgery for hip fracture (HF). Arch Gerontol Geriatr 2015;60:366–71. 10.1016/j.archger.2014.11.008 [DOI] [PubMed] [Google Scholar]
  • 39.Sun T, Wang X, Liu Z, et al. Plasma concentrations of pro- and anti-inflammatory cytokines and outcome prediction in elderly hip fracture patients. Injury 2011;42:707–13. 10.1016/j.injury.2011.01.010 [DOI] [PubMed] [Google Scholar]
  • 40.Frost SA, Nguyen ND, Black DA, et al. Risk factors for in-hospital post-hip fracture mortality. Bone 2011;49:553–8. 10.1016/j.bone.2011.06.002 [DOI] [PubMed] [Google Scholar]
  • 41.Otete H, Deleuran T, Fleming KM, et al. Hip fracture risk in patients with alcoholic cirrhosis: a population-based study using English and Danish data. J Hepatol 2018;69:697–704. 10.1016/j.jhep.2018.04.002 [DOI] [PubMed] [Google Scholar]
  • 42.Vestergaard P, Rejnmark L, Mosekilde L. Increased mortality in patients with a hip fracture-effect of pre-morbid conditions and post-fracture complications. Osteoporos Int 2007;18:1583–93. 10.1007/s00198-007-0403-3 [DOI] [PubMed] [Google Scholar]
  • 43.Jiang HX, Majumdar SR, Dick DA, et al. Development and initial validation of a risk score for predicting in-hospital and 1-year mortality in patients with hip fractures. J Bone Miner Res 2005;20:494–500. 10.1359/JBMR.041133 [DOI] [PubMed] [Google Scholar]
  • 44.Padrón-Monedero A, López-Cuadrado T, Galán I, et al. Effect of comorbidities on the association between age and hospital mortality after fall-related hip fracture in elderly patients. Osteoporos Int 2017;28:1559–68. 10.1007/s00198-017-3926-2 [DOI] [PubMed] [Google Scholar]
  • 45.Bass E, French DD, Bradham DD, et al. Risk-Adjusted mortality rates of elderly veterans with hip fractures. Ann Epidemiol 2007;17:514–9. 10.1016/j.annepidem.2006.12.004 [DOI] [PubMed] [Google Scholar]
  • 46.Keegan MT, Plevak DJ. Preoperative assessment of the patient with liver disease. Am J Gastroenterol 2005;100:2116–27. 10.1111/j.1572-0241.2005.41453.x [DOI] [PubMed] [Google Scholar]
  • 47.Nakchbandi IA. Osteoporosis and fractures in liver disease: relevance, pathogenesis and therapeutic implications. World J Gastroenterol 2014;20:9427–38. 10.3748/wjg.v20.i28.9427 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 48.Montomoli J, Erichsen R, Gammelager H, et al. Liver disease and mortality among patients with hip fracture: a population-based cohort study. Clin Epidemiol 2018;10:991–1000. 10.2147/CLEP.S168237 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49.Penrod JD, Litke A, Hawkes WG, et al. The association of race, gender, and comorbidity with mortality and function after hip fracture. J Gerontol A Biol Sci Med Sci 2008;63:867–72. 10.1093/gerona/63.8.867 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 50.Omari A, Madsen CM, Lauritzen JB, et al. Comorbidity and mortality after hip fracture in nineteen thousand six hundred and eighty two patients aged eighteen to sixty five years in Denmark from 1996 to 2012. Int Orthop 2019;43:2621–7. 10.1007/s00264-019-04323-z [DOI] [PubMed] [Google Scholar]
  • 51.Roche JJW, Wenn RT, Sahota O, et al. Effect of comorbidities and postoperative complications on mortality after hip fracture in elderly people: prospective observational cohort study. BMJ 2005;331:1374. 10.1136/bmj.38643.663843.55 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 52.Dodd AC, Bulka C, Jahangir A, et al. Predictors of 30-day mortality following hip/pelvis fractures. Orthop Traumatol Surg Res 2016;102:707–10. 10.1016/j.otsr.2016.05.016 [DOI] [PubMed] [Google Scholar]
  • 53.Karres J, Heesakkers NA, Ultee JM, et al. Predicting 30-day mortality following hip fracture surgery: evaluation of six risk prediction models. Injury 2015;46:371–7. 10.1016/j.injury.2014.11.004 [DOI] [PubMed] [Google Scholar]
  • 54.Nia A, Popp D, Thalmann G, et al. Predicting 30-day and 180-day mortality in elderly proximal hip fracture patients: evaluation of 4 risk prediction scores at a level I trauma center. Diagnostics (Basel) 2021;11:497. 10.3390/diagnostics11030497 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 55.Ramanathan TS, Moppett IK, Wenn R, et al. Possum scoring for patients with fractured neck of femur † †an Abstract of part of the study was presented at the anaesthetic research Society meeting, Aberdeen, April 2004 and published in British Journal of anaesthesia 2004; 93: 161. British Journal of Anaesthesia 2005;94:430–3. 10.1093/bja/aei064 [DOI] [PubMed] [Google Scholar]
  • 56.Wanjiang F, Xiaobo Z, Xin W, et al. Application of POSSUM and P-POSSUM scores in the risk assessment of elderly hip fracture surgery: systematic review and meta-analysis. J Orthop Surg Res 2022;17:255. 10.1186/s13018-022-03134-0 [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary data

bmjopen-2022-068465supp001.pdf (86.8KB, pdf)

Supplementary data

bmjopen-2022-068465supp002.pdf (127KB, pdf)

Supplementary data

bmjopen-2022-068465supp003.pdf (60KB, pdf)

Supplementary data

bmjopen-2022-068465supp004.pdf (95.1KB, pdf)

Supplementary data

bmjopen-2022-068465supp005.pdf (86.8KB, pdf)

Reviewer comments
Author's manuscript

Data Availability Statement

Data are available on reasonable request. The datasets generated and analysed during the current study are not publicly available due the fact that they constitute an excerpt of research in progress but are available from the corresponding author on reasonable request.


Articles from BMJ Open are provided here courtesy of BMJ Publishing Group

RESOURCES