Abstract
Background
Frailty is an important determinant of health‐care needs and outcomes for people in hospital.
Objectives
To compare characteristics and predictive ability of a multidomain frailty index derived from routine health data (electronic frailty index‐acute hospital; eFI‐AH) with the hospital frailty risk score (HFRS).
Methods
This retrospective study included 6771 patients aged ≥75 years admitted to an Australian metropolitan tertiary referral hospital between October 2019 and September 2020. The eFI‐AH and the HFRS were calculated for each patient and compared with respect to characteristics, agreement, association with age and ability to predict outcomes.
Results
Median eFI‐AH was 0.17 (range 0–0.66) whilst median HFRS was 3.2 (range 0–42.9). Moderate agreement was shown between the tools (Pearson's r 0.61). After adjusting for age and gender, both models had associations with long hospital stay, in‐hospital mortality, unplanned all‐cause readmission and fall‐related readmission. Specifically, the eFI‐AH had the strongest association with in‐hospital mortality (adjusted odds ratio (aOR) 2.81, 95% confidence intervals (CI) 2.49–3.17), whilst the HFRS was most strongly associated with long hospital stay (aOR 1.20, 95% CI 1.18–1.21). Both tools predicted hospital stay >10 days with good discrimination and calibration.
Conclusions
Although the eFI‐AH and the HFRS did not consistently identify the same inpatients as frail, both were associated with adverse outcomes and they had comparable predictive ability for prolonged hospitalisation. These two constructs of frailty may have different implications for clinical practice and health service provision and planning.
Keywords: frail elderly, frailty, hospitals, length of stay, risk assessment
Practice Impact.
The electronic frailty index using data extracted from the hospital electronic medical record (eFI‐AH) and the hospital frailty risk score from ICD‐10 codes (HFRS) identify different acute hospital patients as frail, with comparable prediction of long length‐of‐stay. The eFI‐AH can be adapted for real‐time, automated identification of frail patients for further assessment and specialised care.
1. INTRODUCTION
Frailty, a measure of a person's vulnerability to external stressors, is an important determinant of health‐care needs and outcomes for people in hospital, at the individual and system levels. With increasing frailty, people are less able to respond to the range of stressors experienced in hospital and need specialised multidisciplinary care based on principles of geriatric evaluation and management. 1 Therefore, it is important to identify patients with frailty in the hospital setting, to pave the way for further needs assessment and better care coordination. 2
Frailty assessment for acute hospital inpatients should be valid and should not be onerous to health‐care staff to facilitate wide implementation. 3 Furthermore, frailty should reflect baseline vulnerability across multiple domains, which includes but is not exclusively related to comorbidities rather than severity of acute illness. To inform individual patient care in hospital, frailty needs to be assessed in real time early in the hospital admission. 4 A frailty measure calculated after discharge could be applied to post‐discharge care of the individual and to service provision at a system level. Many frailty measures and acute illness severity may predict adverse global health outcomes, but their interpretation and application will differ. 1 , 5
Numerous validated frailty measures have been developed. 6 The Clinical Frailty Scale (CFS) is a quick screening tool that requires clinical observation by clinicians, which might be subject to interrater reliability and may not always be available at hospital admission. 7 , 8 , 9 The Fried frailty phenotype (FP) assesses the presence of five physical frailty criteria: weight loss, exhaustion, grip strength, walking speed and physical activity, 10 which may be heavily influenced by acute severe illness requiring hospitalisation. The FP was designed as a categorical measure (i.e., robust, pre‐frail, or frail), which limits its predictive validity. 11 Aiming for higher predictive validity in hospital inpatients, we therefore focused on two well validated inpatient frailty measures, the Hospital Frailty Risk Score (HFRS) and the Frailty Index (FI), which differ conceptually and pragmatically. The HFRS uses ICD‐10 (International Statistical Classification of Disease and Related Health Problems, 10th revision) codes to calculate scores. 12 Whilst the HFRS has the advantage of being calculated from administrative data that is routinely coded on hospital discharge without additional burden on clinical staff, 13 it is predominantly a co‐morbidity score that is highly influenced by the severity and complexity of the acute illness. A multidomain frailty index, using the well validated cumulative deficit model proposed by Rockwood and colleagues that includes domains across physical, cognitive and social functioning, nutrition and diagnoses, 14 may give a better estimate of underlying vulnerability, but has previously been limited by feasibility to calculate at scale in hospital patients. While the HFRS is reliant on the ICD‐10 coding after discharge so it can only be applied retrospectively; depending on the feasibility of extracting multidomain deficits, a frailty index could be applied in real time during admission or after discharge.
A multidomain frailty index was recently developed using data that was already routinely collected to fulfil the clinical care standards for Australian hospital accreditation (electronic frailty index – acute hospital, eFI‐AH). 15 We hypothesised that the eFI‐AH would differ from the HFRS; and that both would be associated with health outcomes. The aim of this study was to compare the distribution and validity of the eFI‐AH and the HFRS for older people admitted to an Australian metropolitan tertiary referral hospital. Specifically, we aimed to compare: (1) the characteristics of the eFI‐AH and the HFRS, (2) the prevalence of frail patients measured by the tools, (3) degree of agreement in frailty ratings between the tools and correlations of each tool with CCI, (4) the association of the tools with chronological age and negative outcomes (i.e., in‐hospital mortality, long hospital stay, unplanned all‐cause readmission and fall‐related readmission), and (5) the ability to predict the risk of these outcomes, including estimating cut‐offs.
2. METHODS
2.1. Study design, setting and participants
This study was a retrospective observational cohort study using data electronically extracted from routinely collected data from the hospital electronic medical record (eMR). The site of this study was Royal North Shore Hospital (RNSH), which is one of the 11 principal tertiary referral hospitals located in major cities in New South Wales (NSW) Australia. This hospital provides a comprehensive range of complex services. Patient characteristics such as age, gender, education level and comorbidities amongst the 11 hospitals are shown in Appendix S1. Inclusion criteria were patients aged 75 years or older (on the day of the index admission) admitted to the RNSH between October 1, 2019 and September 30, 2020. Exclusion criterion was patients without a valid eFI‐AH (see below). For patients with multiple admissions during the study period, frailty was only calculated for the first admission. Ethics approval was obtained from the Northern Sydney Local Health District Human Research Ethics Committee (reference: 2020/ETH01643). A waiver of consent was granted for the study.
2.2. Calculation of the eFI‐AH and the HFRS
The eFI‐AH consists of 37 deficits across multiple domains as previously reported. 15 Multidomain deficits relating to physical and cognitive functioning, and outcome measures such as delirium, falls and pressure ulcer incidence were derived from routinely completed risk screening and assessments in the hospital setting including: the Brief Cognitive Screen Six Item Screener, 16 Confusion Assessment Method, 17 Waterlow Scale—Pressure Area Risk Assessment, 18 Ontario Modified Stratify (OMS) Falls Risk Screen 19 and speech pathology review. The deficits relating to medical diagnoses included in our eFI‐AH were identified using data from ICD‐10‐AM or the Anatomical Therapeutic Chemical classification of the World Health Organization 20 and pathology results. Deficits were selected and classified informed by expertise in geriatric medicine and frailty indices within the study team. Because a frailty index with fewer than 30 items is not considered a valid frailty measure for predicting adverse outcomes, the eFI‐AH was only calculated for patients with ≥30 completed deficits. 21 The eFI‐AH was retrospectively calculated as the number of reported deficits divided by the total number of deficits evaluated, for example 6/37 = 0.16. Missing eFI‐AH items were managed using the delta denominator method, for example 5/30 = 0.17. 3 Patients were classified into frailty groups: robust (eFI‐AH < 0.05), prefrail (0.05 ≤ eFI‐AH < 0.25) or frail (0.25 ≤ eFI‐AH), based on previously validated cut‐off points. 14 , 22 Of the 37 deficits of the eFI‐AH, 13 are dependent on the ICD‐10 Australian Modification (ICD‐10‐AM) codes and 24 are assessed during hospitalisation. These 24 components can be measured more than once during the admission whilst the 13 ICD‐10‐AM‐dependent components are coded once after discharge. Therefore, the eFI‐AH at the first instance, when the maximum number of items to be evaluated was reached during the hospitalisation, was designated as the eFI‐AH at admission. 21 Likewise, the eFI‐AH closest to the time of discharge was used as an eFI‐AH at discharge.
The HFRS consists of 109 frailty‐related ICD‐10 codes. Each code is assigned a weighted point value. A HFRS is calculated by summing up the point values, ranging from 0 to 99. 13 The same point values assigned to the 109 ICD‐10 codes were adapted to the ICD‐10‐AM code. Since the ICD‐10‐AM codes were assigned after hospital discharge, the HFRS was retrospectively calculated once per admission. Patients were grouped into a low risk (HFRS < 5), intermediate risk (5 ≤ HFRS < 15) or high risk (15 ≤ HFRS), based on previously validated cut‐off points. 13
2.3. Characteristics of the eFI‐AH and the HFRS
The proportion of patients with less than or equal to 30 deficits evaluated (i.e., those in whom the eFI‐AH could be measured) was calculated. The characteristics of patients with a valid eFI‐AH were compared with those with less than 30 deficits items. The cohort of patients in whom the analyses of both the eFI‐AH and the HFRS were conducted was limited to those with a valid eFI‐AH.
A quantile‐quantile (Q‐Q) plot and the Kolmogorov–Smirnov test were used to ascertain whether the eFI‐AH and the HFRS followed a normal distribution or a gamma distribution. 23 A quantile regression analysis was employed to identify the relationship between age and the frailty assessments at the quantile levels of 0.5 and 0.99. 24 Additionally, to evaluate the impact of a given variable on the eFI‐AH, an iterative, re‐sampling process, similar to ‘bootstrapping’, was used following the procedure described by Rockwood et al. 21 , 25 Satisfying the minimum number of deficit items (i.e., 30 deficits), we randomly selected 80% of the 37 deficits (37 × 0.8 ≈ 30 deficits) without replacement, and calculated the eFI‐AH for each sample. Using a quantile regression, the relationship between age and the eFI‐AH at the 50th quantile was evaluated with the slope and intercept recorded. This procedure was repeated 1000 times to generate 1000 samples.
For calculating the degree of agreement in frailty ratings between the tools, patients were classified as either frail or non‐frail using each scale. For this study, a threshold of 0.25 for the eFI‐AH and 15 for the HFRS was used to indicate frailty. 13 Cohen's kappa coefficients with their 95% confidence intervals (CIs) were calculated. Additionally, Pearson's correlation coefficient was used to describe the association between the continuous versions of the tools. Correlations between both tools and the Charlson Comorbidity Index (CCI) were also assessed. 26
To evaluate the validity of the operational threshold (i.e., 0.25 for eFI‐AH and 15 for HFRS) in our study population, optimal threshold values were derived from the area under the receiver operating characteristic curve (ROC‐AUC) analysis using the Youden Index.
2.4. Analyses for association between the eFI‐AH and the HFRS and 4 outcomes
Associations of the frailty assessment tools with the following four outcomes were evaluated: in‐hospital mortality, long hospital stay (>10 days), unplanned all‐cause readmission and fall‐related readmission within 28 days from hospital discharge. These outcomes were extracted from the electronic medical records. Consequently, readmission outcomes could only be assessed within the local health district and may be an underestimate. The eFI‐AH at time of admission was used for in‐hospital mortality and long hospital stay while the eFI‐AH at time of discharge was used for all‐cause and fall‐related readmission. Patients who died during the admission and patients with the lack of 28 days follow‐up period were excluded only when evaluating the associations of the tools with all‐cause and fall‐related readmission.
2.5. Statistical analyses
Data were summarised using descriptive methods such as median and interquartile range (IQR). Multicollinearity was tested using the variance inflation factor (VIF <10). Quantile regression with bootstrapped standard errors was used to evaluate the relationship of age with the tools at selected quantiles of the frailty scores, 0.5 and 0.99. 27 Further details are given in Appendix S2. Logistic regression was conducted to evaluate the association between prognostic variables and each outcome. Models were adjusted for age and gender based on the existing knowledge of risk factors for the outcomes. 28 Analyses of the unadjusted model and age‐, gender‐ and CCI‐ adjusted model were also conducted. Model performance was evaluated through discrimination and calibration.
To assess the discriminative ability of the eFI‐AH and the HFRS models, ROC‐AUC and the area under the precision recall curve (PR‐AUC) were measured using the 10‐fold cross‐validation method. 29 Additionally, sensitivity, specificity, positive predictive value (PPV), false positive rate (FPR), negative predictive value (NPV) and accuracy were averaged across the 10 folds at the optimal cut‐off point at which the Youden Index was maximised. Score calibration was assessed using the Hosmer‐Lemeshow test and through comparing the slopes and intercepts across deciles of the risk of outcomes. 30 Data manipulation and statistical analyses were performed using Python version 3.8.8 (Python Software Foundation) and R version 3.6.1 (R Foundation). Two‐sided p‐values <0.05 were considered statistically significant.
3. RESULTS
Of a total of 10,141 patients who met the inclusion criteria, 3370 patients with fewer than 30 items evaluated were excluded (Appendix S3). The excluded 3370 patients had a shorter length of stay compared with the included 6771 patients (median 0.2 days, IQR 0.1–0.4 and 4.1 days, IQR 2.0–8.0, respectively; Appendix S4). The prevalence of frailty‐related ICD‐10‐AM codes used to calculate HFRS amongst patients included in and excluded from the analysis is shown in Appendix S5. For patients admitted for more than 24 hours, 97% of them were included in this study. The median age of the included cohort was 83.5 years and 52% were female (Table 1). The impact of the COVID‐19 pandemic was rather minimal during the study period, with only 12 patients in the cohort hospitalised with COVID‐19, although overall hospital admissions in 2019/20 were 3% lower than 2018/19.
TABLE 1.
Characteristics of the study population (n = 6771)
| Count (%) | eFI‐AH | HFRS | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Median (IQR) | Robust < 0.1 (%) | Prefrail 0.1 ≤ eFI‐AH < 0.25 (%) | Frail ≥ 0.25 (%) | Median (IQR) | Low risk < 5 (%) | Intermediate risk 5 ≤ HFRS < 15 (%) | High risk ≥ 15 (%) | ||
| Age (years) | |||||||||
| 75–79 | 2083 (31) | 0.14 (0.08–0.20) | 697 (34) | 1072 (52) | 314 (15) | 2.0 (0.0–5.8) | 1484 (71) | 511 (24) | 88 (4) |
| 80–84 | 1886 (28) | 0.16 (0.10–0.24) | 478 (26) | 978 (52) | 430 (23) | 3.0 (0.0–7.3) | 1186 (63) | 582 (31) | 118 (6) |
| 85–89 | 1537 (23) | 0.19 (0.12–0.27) | 231 (15) | 854 (56) | 452 (29) | 3.7 (0.8–8.1) | 905 (59) | 515 (34) | 117 (8) |
| 90–94 | 938 (14) | 0.23 (0.15–0.31) | 97 (10) | 456 (49) | 385 (41) | 5.5 (1.8–10.1) | 430 (46) | 410 (44) | 98 (10) |
| ≥95 | 327 (5) | 0.24 (0.18–0.32) | 18 (6) | 150 (46) | 159 (49) | 5.5 (2.3–9.9) | 148 (45) | 149 (46) | 30 (9) |
| Gender | |||||||||
| Female | 3535 (52) | 0.18 (0.11–0.26) | 750 (21) | 1842 (52) | 943 (27) | 3.4 (0.5–8.0) | 2116 (60) | 1195 (34) | 224 (6) |
| Male | 3236 (48) | 0.16 (0.11–0.24) | 771 (24) | 1668 (52) | 797 (24) | 3.0 (0.0–7.3) | 2037 (63) | 972 (30) | 227 (7) |
| CCI | |||||||||
| 0 | 3408 (50) | 0.14 (0.08–0.20) | 1188 (35) | 1723 (51) | 497 (15) | 2.3 (0.0–6.0) | 2384 (70) | 924 (27) | 100 (3) |
| 1 | 1473 (22) | 0.2 (0.14–0.28) | 175 (12) | 799 (54) | 499 (34) | 4.3 (1.1–9.4) | 804 (55) | 514 (35) | 155 (10) |
| 2 | 972 (14) | 0.2 (0.14–0.28) | 111 (11) | 539 (56) | 322 (33) | 4.1 (0.7–8.4) | 545 (56) | 350 (36) | 77 (8) |
| 3 | 293 (4) | 0.26 (0.19–0.35) | 7 (2) | 137 (47) | 149 (51) | 6.4 (2.6–12.8) | 124 (42) | 125 (43) | 44 (15) |
| 4 | 165 (2) | 0.26 (0.19–0.35) | 3 (2) | 77 (47) | 85 (52) | 7.1 (3.6–11.2) | 59 (36) | 85 (52) | 21 (13) |
| 5 | 291 (4) | 0.2 (0.14–0.30) | 31 (11) | 160 (55) | 100 (34) | 4.6 (1.6–9.3) | 157 (54) | 110 (38) | 24 (8) |
| 6 | 105 (2) | 0.24 (0.18–0.32) | 5 (5) | 52 (50) | 48 (46) | 4.5 (2.3–8.8) | 58 (55) | 34 (32) | 13 (12) |
| ≥7 | 64 (1) | 0.3 (0.23–0.38) | 1 (2) | 23 (36) | 40 (62) | 8.8 (4.2–15.3) | 22 (34) | 25 (39) | 17 (27) |
| LOS (days) | |||||||||
| Q1: 0‐ < 1.9 | 1694 (25) | 0.11 (0.06–0.16) | 811 (48) | 748 (44) | 135 (8) | 0.0 (0.0–2.0) | 1530 (90) | 159 (9) | 5 (<1) |
| Q2: 1.9‐ < 4.1 | 1692 (25) | 0.16 (0.11–0.23) | 400 (24) | 959 (57) | 333 (20) | 2.3 (0.4–5.6) | 1199 (71) | 454 (27) | 39 (2) |
| Q3: 4.1‐ < 8.0 | 1692 (25) | 0.19 (0.14–0.27) | 205 (12) | 995 (59) | 492 (29) | 4.5 (1.8–8.2) | 908 (54) | 691 (41) | 93 (6) |
| Q4: ≥8.0 | 1693 (25) | 0.24 (0.18–0.32) | 105 (6) | 808 (48) | 780 (46) | 8.1 (4.1–13.2) | 516 (30) | 863 (51) | 314 (18) |
| Overall, No. | 6771 (100) | 0.17 (0.11–0.26) | 1521 (22) | 3510 (52) | 1740 (26) | 3.2 (0.4–7.7) | 4153 (61) | 2167 (32) | 451 (7) |
Abbreviations: CCI, Charlson Comorbidity Index; eFI‐AH, electronic‐Frailty Index – Acute Hospital; HFRS, Hospital Frailty Risk Score; IQR, Interquartile range; LOS, length of stay; Q, quartile.
Of 6771 patients, 5693 patients (84%) had all 37 deficits evaluated. Of 37 deficits, 19 deficits were evaluated for all patients (Appendix S6). Of the 37 deficits, the highest prevalence was skin type visual risk (71%), followed by arrhythmia and heart disease (39%, Appendix S7). The distribution of the eFI‐AH was approximated by a gamma distribution, especially when stratified by age groups (Figure 1, Appendix S8). The eFI‐AH classified 23% of patients as robust, 52% as prefrail and 26% as frail. The eFI‐AH at discharge was slightly lower than that at admission (median 0.17, range 0–0.66 and 99th percentile 0.49 on admission and median 0.16, range 0–0.64 and 99th percentile 0.49 on discharge, p < 0.01). Random sampling of the eFI‐AH showed that when 80% of 37 items were randomly sampled, the slope was 0.0055 (95% CI 0.0031–0.0080), indicating there was little sensitivity as to which variables were included in the eFI‐AH construction (Appendix S9).
FIGURE 1.
Histograms of the electronic‐Frailty Index – Acute Hospital (eFI‐AH) at admission and the Hospital Frailty Risk Score (HFRS; n = 6771).
The HFRS for the cohort ranged from 0 to 42.9 with a median of 3.2 (IQR 0.4–7.7 and 99th percentile 23.4). More than half of the patients were classified as low risk (61%), approximately a third as intermediate risk (32%) and 7% as high risk. The distribution of the HFRS was highly skewed to the right, approximated by a gamma distribution especially when stratified by age groups (Appendix S10). The highest prevalence of HFRS‐contributing ICD‐10‐AM codes was N17 (acute renal failure, 14%), followed by E87 (other disorders of fluid, electrolyte and acid–base balance, 14%) and I95 (hypotension, 14%, Appendix S5).
In terms of the degree of agreement, both kappa statistics and Pearson's correlation coefficient showed moderate agreement between the tools (kappa = 0.42, 95% CI 0.40–0.44, Pearson's r = 0.61, 95% CI 0.59–0.62). Both the eFI‐AH and the HFRS displayed weak correlations with the CCI (Pearson's r = 0.32, 95% CI 0.30–0.34, and Pearson's r = 0.21, 95% CI 0.18–0.23, respectively).
Optimal threshold values for the eFI‐AH were different depending on the outcomes of interest; 0.26 for in‐hospital mortality, 0.18 for long hospital stay, 0.13 for all‐cause readmission and 0.15 for fall‐related readmission. Optimal threshold values for the HFRS were 4.40 for in‐hospital mortality, 4.50 for long length of stay, 3.20 for all‐cause readmission and 4.70 for fall‐related readmission.
Table 2 shows the associations between the tools and the four outcomes. All variables had a VIF <10, suggesting that multicollinearity was not a major issue. After adjusting for age and gender, both the eFI‐AH and the HFRS models had significant associations with all outcomes. Specifically, the eFI‐AH had the strongest association with in‐hospital mortality (adjusted odds ratio (aOR) 2.95, 95% CI 2.63–3.31) whilst the HFRS with long hospital stay (aOR 1.21, 95% CI 1.20–1.23). Unadjusted models and age‐, gender‐ and CCI‐adjusted models were also associated with all outcomes (Appendices S11 and S12). The CCI was statistically associated with all outcomes. However, the predictive ability of the CCI for the outcomes was lower than the eFI‐AH and the HFRS (Appendix S13).
TABLE 2.
Association, discriminative ability and calibration of the eFI‐AH and the HFRS for four outcomes
| Items | Unplanned all‐cause readmission within 28 days b (n = 666/5938) | Fall‐related readmission within 28 days b (n = 121/5938) | Length of stay (>10 days; n = 1240/6771) | In‐hospital mortality (n = 255/6771) | ||||
|---|---|---|---|---|---|---|---|---|
| eFI‐AH | HFRS | eFI‐AH | HFRS | eFI‐AH | HFRS | eFI‐AH | HFRS | |
| Adjusted Odds Ratio (95% CI) a | 1.11 (1.02–1.20)** | 1.02 (1.01–1.04)** | 1.37 (1.16–1.62)*** | 1.05 (1.02–1.08)*** | 2.04 (1.91–2.17)*** | 1.20 (1.18–1.21)*** | 2.81 (2.49–3.17)*** | 1.06 (1.05–1.08)*** |
| ROC‐AUC | 0.58 | 0.59 | 0.65 | 0.66 | 0.75 | 0.81 | 0.87 | 0.77 |
| PR‐AUC | 0.15 | 0.15 | 0.04 | 0.04 | 0.39 | 0.51 | 0.18 | 0.1 |
| TPR (Recall) c | 0.4 | 0.31 | 0.65 | 0.62 | 0.74 | 0.81 | 0.85 | 0.82 |
| FPR c | 0.31 | 0.22 | 0.4 | 0.37 | 0.35 | 0.34 | 0.24 | 0.4 |
| TNR (Specificity) c | 0.69 | 0.78 | 0.6 | 0.63 | 0.65 | 0.66 | 0.76 | 0.6 |
| Accuracy c | 0.65 | 0.72 | 0.6 | 0.63 | 0.67 | 0.69 | 0.76 | 0.61 |
| Precision (PPV) c | 0.17 | 0.19 | 0.03 | 0.03 | 0.32 | 0.35 | 0.12 | 0.08 |
| NPV c | 0.88 | 0.88 | 0.99 | 0.99 | 0.92 | 0.94 | 0.99 | 0.99 |
| F1‐score c | 0.24 | 0.24 | 0.06 | 0.06 | 0.45 | 0.49 | 0.21 | 0.14 |
| Calibration intercept | 0.01 | 0.00 | 0.00 | 0.00 | −0.01 | 0.00 | −0.01 | 0.00 |
| Calibration slope | 0.94 | 0.98 | 0.86 | 0.84 | 1.05 | 0.99 | 1.23 | 0.93 |
| p Value d | 0.47 | 0.17 | 0.009 | 0.30 | < 0.001 | < 0.001 | < 0.001 | 0.02 |
Abbreviations: eFI‐AH, electronic‐Frailty Index – Acute Hospital; FPR, false positive rate; HFRS, Hospital Frailty Risk Score; NPV, negative predictive value; PPV, positive predictive value; PR‐AUC, area under the precision recall curve; ROC‐AUC, area under the receiver operating characteristic curve; TNR, true negative rate; TPR, true positive rate.
The logistic regression model was adjusted for age and gender. Scale of eFI‐AH was changed by multiplying by 10.
255 patients who died during the admission and 578 patients with lack of 28 days follow‐up period were only excluded from analyses of readmission. p‐value: *<0.05, **<0.01, ***<0.001.
Each was averaged across the 10 folds at the optimal cut‐off point at which the Youden Index was maximised.
Hosmer‐Lemeshow test.
Table 2 and Figure 2 summarise the discriminatory ability of the eFI‐AH and the HFRS models for the four outcomes. Both models for predicting long hospital stay showed good discrimination (AUC‐ROC = 0.74, 0.79 and PR‐AUC = 0.39, 0.49, respectively) and calibration (slope = 1.09, 0.99 and intercept = −0.02, 0 respectively, Figure 3). In the highest decile frailty group, mean predicted risk was 47% for the eFI‐AH model and 62% for the HFRS model, whilst in the lowest decile group, 6% for the eFI‐AH and 5% for the HFRS model, demonstrating a wide range of predicted risk probabilities. The eFI‐AH model for in‐hospital mortality had the highest ROC‐AUC (0.86) of all models. However, PR‐AUC was only 0.18 with narrow range of predicted risk due to the low prevalence of in‐hospital mortality (mean predicted risk in the lowest decile 0.01% ‐ the highest decile 16%). The Hosmer‐Lemeshow statistics for long hospital stay and for in‐hospital mortality were significant (p < 0.05) for both models, indicating that they may not be an optimal fit. Both models discriminated very weakly between patients with and without all‐cause readmission as well as fall‐related readmission.
FIGURE 2.
Receiver operating characteristic curves and precision recall curves with their respective areas under the curves. eFI‐AH, electronic‐Frailty Index – Acute Hospital; HFRS, Hospital Frailty Risk Score. Each model was adjusted for age and gender. Area under the curve for each frailty measure and outcome is reported in the legend.
FIGURE 3.
Calibration plots for the eFI‐AH and the HFRS with four outcomes. eFI‐AH, electronic‐Frailty Index – Acute Hospital; HFRS, Hospital Frailty Risk Score. Each model was adjusted for age and gender.
4. DISCUSSION
The present study confirmed that although the eFI‐AH and the HFRS did not identify the same inpatients as frail, they had comparable ability to predict the risk of the outcomes of interest. Specifically, both had good predictive ability for long hospital stay but not the other outcomes. Considering that longer hospital stays are recognised to be responsible for considerable additional health‐care costs and resource utilisation associated with frailty, the retrospective use of the eFI‐AH and/or the HFRS could be beneficial at the level of the health‐care system to inform resourcing of patient care and understand variability.
The degree of agreement between the tools was better than that in the original HFRS study (Pearson = 0.41, kappa = 0.30 against the Rockwood Frailty Index). 13 Compared with the eFI‐AH, the distribution of the HFRS was highly positively skewed with narrow IQR. The difference of the distributions between the tools can affect the proportion of patients identified as frail. If the eFI‐AH or the HFRS are to be used for estimating the prevalence of patients with frailty for health‐care resource allocation and utilisation, the difference of the distribution in the frailty measured using the tools would be an important consideration.
Both the cumulative deficit model and the HFRS have been validated in various countries. Our results were similar to previous studies for association and predictive ability of the Frailty Index 31 , 32 and the HFRS 13 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 for long hospital stay and unplanned all‐cause readmissions, with similar ROC‐AUC. The optimal threshold values for both tools were different for each outcome, suggesting that thresholds need to be carefully set according to the outcome that is being investigated. This study also showed the importance of reporting PR‐AUC when the outcome of interest has low prevalence. For example, although both tools were associated with in‐hospital mortality and had good ROC‐AUC, their discriminative ability judging from PR‐AUC was low with narrow range of predicted risk probability. Unlike the precision and recall that focus on the true positive cases (the minority category), FPR focuses on the true negative cases (the majority category). In this study, prevalence of in‐hospital mortality was only 4%, therefore, the use of ROC‐AUC, as in most previous studies of the HFRS, provides an optimistic view of the discriminative ability. A comprehensive assessment is required to ensure the predictive ability of frailty measurement tools from the perspective of validity and comparability, as well as for consideration of clinical utility at an individual patient level.
There are several limitations that must be addressed in future studies. First, we only included patients aged 75 years or over, and as such, the results may not be applicable to younger patients. Secondly, this single‐site study lacks external validation for assessing generalisability in different populations from a variety of services across multiple hospitals. Interestingly, a population‐based study including patients aged ≥50 years admitted for surgery to hospitals in NSW, Australia, also found associations of the HFRS with long hospital stay and readmission with poor discriminative ability (long hospital stay: ROC‐AUC 0.61, readmission: ROC‐AUC 0.64). 28 Thirdly, our study was limited to those in whom there was adequate data to calculate eFI‐AH (67% of all patients, 97% of patients admitted for >24 h), and thus may not be representative of all patients aged over 75 years in our hospital. Another limitation for clinical utility of the frailty assessments tested is that neither the eFI‐AH and the HFRS can be calculated during hospitalisation because the ICD‐10‐AM codes are assigned to patients after discharge by professional hospital coders. Given the eFI‐AH includes 13 deficits that require the ICD‐10‐AM codes, alternative variables could be explored to enable real‐time identification of frail patients for further assessment and prioritisation for specialised care. Generalisability to Australian hospitals nationally would be feasible, since all deficits are based on data required for hospitals to fulfil the National Safety and Quality Health Care Standards. However, specific validated risk screening tools used are variable, supported by state‐specific eMRs in state‐run public hospitals. Data extraction to calculate some deficits in the eFI‐AH (e.g., falls risk, pressure area risk, nutrition) would require review for each local health district and modifications at some.
5. CONCLUSIONS AND IMPLICATIONS
This study confirmed that the eFI‐AH and the HFRS are both valid measures of frailty that can be feasibly calculated at scale from routine data. Although they identified different groups of inpatients as frail, both were associated with age, long hospital stay, mortality, all‐cause readmission and fall‐related readmission, which may be applicable at the level of health service planning. They had comparable predictive ability for long‐term hospitalisation, but did not accurately predict in‐hospital mortality, all‐cause unplanned readmission and fall‐related readmission, which may limit utility for guiding individual patient care. Adaptation of the eFI‐AH to use data extracted in real time from the eMR would facilitate its use in individual patient care, which requires digital infrastructure planning moving forward. Future studies could assess the association of these frailty measures with other measures of vulnerability such as adverse events and functional recovery, as well as with responsiveness to multidisciplinary geriatric care. This will provide a deeper understanding of differences in the constructs and application to clinical practice and health service provision and planning.
FUNDING INFORMATION
This study was supported by the Australian National Health and Medical Research Council Targeted Call for Research into Frailty in Hospital Care (APP 1174447). The funding sources had no involvement in the design, analysis or writing of the paper.
CONFLICTS OF INTEREST
No conflicts of interest declared.
Supporting information
Appendix S1
ACKNOWLEDGEMENTS
The authors thank Michael Steuernagel and Seven Guney, ICT department, Northern Sydney Local Health District, for their contribution to data extraction for this work. Open access publishing facilitated by The University of Sydney, as part of the Wiley ‐ The University of Sydney agreement via the Council of Australian University Librarians.
Fujita K, Lo SY, Hubbard RE, Gnjidic D, Hilmer SN. Comparison of a multidomain frailty index from routine health data with the hospital frailty risk score in older patients in an Australian hospital. Australas J Ageing. 2023;42:480‐490. doi: 10.1111/ajag.13162
DATA AVAILABILITY STATEMENT
The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions.
REFERENCES
- 1. Clegg A, Young J, Iliffe S, Rikkert MO, Rockwood K. Frailty in elderly people. Lancet. 2013;381(9868):752‐762. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Callahan KE. The future of frailty: opportunity is knocking. J Am Geriatr Soc. 2022;70(1):78‐80. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Morley JEMBB, Vellas BMD, Abellan van Kan GM, et al. Frailty consensus: a call to action. J Am Med Dir Assoc. 2013;14(6):392‐397. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Drubbel I, Numans ME, Kranenburg G, Bleijenberg N, de Wit NJ, Schuurmans MJ. Screening for frailty in primary care: a systematic review of the psychometric properties of the frailty index in community‐dwelling older people. BMC Geriatr. 2014;14:27. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Theou O, Squires E, Mallery K, et al. What do we know about frailty in the acute care setting? A scoping review. BMC Geriatr. 2018;18(1):139. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Hilmer S, Hubbard RE. Where next with frailty risk scores in hospital populations? Age Ageing. 2022;51(1):afab203. [DOI] [PubMed] [Google Scholar]
- 7. Rockwood K, Song X, MacKnight C, et al. A global clinical measure of fitness and frailty in elderly people. Can Med Assoc J. 2005;173(5):489‐495. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Surkan M, Rajabali N, Bagshaw SM, Wang X, Rolfson D. Interrater reliability of the clinical frailty scale by geriatrician and intensivist in patients admitted to the intensive care unit. Can Geriatr J. 2020;23(3):235‐241. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Pugh RJ, Ellison A, Pye K, et al. Feasibility and reliability of frailty assessment in the critically ill: a systematic review. Crit Care. 2018;22(1):49. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Fried LP, Tangen CM, Walston J, et al. Frailty in older adults: evidence for a phenotype. J Gerontol A Biol Sci Med Sci. 2001;56(3):M146‐M156. [DOI] [PubMed] [Google Scholar]
- 11. Kulminski AM, Ukraintseva SV, Kulminskaya IV, Arbeev KG, Land K, Yashin AI. Cumulative deficits better characterize susceptibility to death in elderly people than phenotypic frailty: lessons from the cardiovascular health study. J Am Geriatr Soc. 2008;56(5):898‐903. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. The World Health Organization . List of Official International Statistical Classification of Diseases and Related Health Problems 10th Revision (ICD‐10) Updates. https://www.who.int/standards/classifications/classification‐of‐diseases/list‐of‐official‐icd‐10‐updates. Accessed October 10, 2022.
- 13. Gilbert T, Neuburger J, Kraindler J, et al. Development and validation of a hospital frailty risk score focusing on older people in acute care settings using electronic hospital records: an observational study. Lancet. 2018;391(10132):1775‐1782. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Rockwood K, Andrew M, Mitnitski A. A comparison of two approaches to measuring frailty in elderly people. J Gerontol A Biol Sci Med Sci. 2007;62(7):738‐743. [DOI] [PubMed] [Google Scholar]
- 15. Lo SY, Zhang M, Hubbard RE, Gnjidic D, Redston MR, Hilmer SN. Development and validation of a frailty index based on data routinely collected across multiple domains in NSW hospitals. Australas J Ageing. 2021;40(2):184‐194. [DOI] [PubMed] [Google Scholar]
- 16. Callahan CM, Unverzagt FW, Hui SL, Perkins AJ, Hendrie HC. Six‐item screener to identify cognitive impairment among potential subjects for clinical research. Med Care. 2002;40(9):771‐781. [DOI] [PubMed] [Google Scholar]
- 17. Inouye SK, van Dyck CH, Alessi CA, Balkin S, Siegal AP, Horwitz RI. Clarifying confusion: the confusion assessment method. A new method for detection of delirium. Ann Intern Med. 1990;113(12):941‐948. [DOI] [PubMed] [Google Scholar]
- 18. Waterlow J. Pressure sores: a risk assessment card. Nurs Times. 1985;81(48):49‐55. [PubMed] [Google Scholar]
- 19. Papaioannou A, Parkinson W, Cook R, Ferko N, Coker E, Adachi JD. Prediction of falls using a risk assessment tool in the acute care setting. BMC Med. 2004;2(1):1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. WHO Collaborating Centre for Drug Statistics Methodology . Structure and principles: The WHO Anatomical Therapeutic Chemical (ATC) Classification System. https://www.whocc.no/atc/structure_and_principles/. Accessed October 10, 2022.
- 21. Searle SD, Mitnitski A, Gahbauer EA, Gill TM, Rockwood K. A standard procedure for creating a frailty index. BMC Geriatr. 2008;8(1):24. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Blodgett J, Theou O, Kirkland S, Andreou P, Rockwood K. Frailty in NHANES: comparing the frailty index and phenotype. Arch Gerontol Geriatr. 2015;60(3):464‐470. [DOI] [PubMed] [Google Scholar]
- 23. Mitnitski AB, Mogilner AJ, Rockwood K. Accumulation of deficits as a proxy measure of aging. ScientificWorldJournal. 2001;1:323‐336. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Koenker R, Bassett G. Regression Quantiles. Econometrica. 1978;46(1):33‐50. [Google Scholar]
- 25. Hubbard RE, Peel NM, Samanta M, et al. Derivation of a frailty index from the interRAI acute care instrument. BMC Geriatr. 2015;15(1):27. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. Quan H, Li B, Couris CM, et al. Updating and validating the Charlson comorbidity index and score for risk adjustment in hospital discharge abstracts using data from 6 countries. Am J Epidemiol. 2011;173(6):676‐682. [DOI] [PubMed] [Google Scholar]
- 27. Hahn J. Bootstrapping Quantile Regression Estimators. Economet Theor. 1995;11(1):105‐121. [Google Scholar]
- 28. Harvey LA, Toson B, Norris C, Harris IA, Gandy RC, Close JJCT. Does identifying frailty from ICD‐10 coded data on hospital admission improve prediction of adverse outcomes in older surgical patients? A population‐based study. Age Ageing. 2021;50(3):802‐808. [DOI] [PubMed] [Google Scholar]
- 29. Saito T, Rehmsmeier M. The precision‐recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS One. 2015;10(3):e0118432. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30. Stevens RJ, Poppe KK. Validation of clinical prediction models: what does the “calibration slope” really measure? J Clin Epidemiol. 2020;118:93‐99. [DOI] [PubMed] [Google Scholar]
- 31. Kerminen H, Huhtala H, Jäntti P, Valvanne J, Jämsen E. Frailty index and functional level upon admission predict hospital outcomes: an interRAI‐based cohort study of older patients in post‐acute care hospitals. BMC Geriatr. 2020;20(1):160. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32. Hubbard RE, Peel NM, Samanta M, Gray LC, Mitnitski A, Rockwood K. Frailty status at admission to hospital predicts multiple adverse outcomes. Age Ageing. 2017;46(5):801‐806. [DOI] [PubMed] [Google Scholar]
- 33. McAlister F, van Walraven C. External validation of the hospital frailty risk score and comparison with the hospital‐patient one‐year mortality risk score to predict outcomes in elderly hospitalised patients: a retrospective cohort study. BMJ Qual Saf. 2019;28(4):284‐288. [DOI] [PubMed] [Google Scholar]
- 34. Eckart A, Hauser SI, Haubitz S, et al. Validation of the hospital frailty risk score in a tertiary care hospital in Switzerland: results of a prospective, observational study. BMJ Open. 2019;9(1):e026923. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35. McAlister FA, Savu A, Ezekowitz JA, Armstrong PW, Kaul P. The hospital frailty risk score in patients with heart failure is strongly associated with outcomes but less so with pharmacotherapy. J Intern Med. 2020;287(3):322‐332. [DOI] [PubMed] [Google Scholar]
- 36. Hannah TC, Neifert SN, Caridi JM, et al. Utility of the hospital frailty risk score for predicting adverse outcomes in degenerative spine surgery cohorts. Neurosurgery. 2020;87(6):1223‐1230. [DOI] [PubMed] [Google Scholar]
- 37. Shebeshi DS, Dolja‐Gore X, Byles J. Validation of hospital frailty risk score to predict hospital use in older people: evidence from the Australian longitudinal study on Women's health. Arch Gerontol Geriatr. 2021;92:104282. [DOI] [PubMed] [Google Scholar]
- 38. Hollinghurst J, Housley G, Watkins A, Clegg A, Gilbert T, Conroy SP. A comparison of two national frailty scoring systems. Age Ageing. 2021;50(4):1208‐1214. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39. Aitken SJ, Lujic S, Randall DA, Noguchi N, Naganathan V, Blyth FM. Predicting outcomes in older patients undergoing vascular surgery using the hospital frailty risk score. Br J Surg. 2021;108(6):659‐666. [DOI] [PubMed] [Google Scholar]
- 40. Gilbert T, Cordier Q, Polazzi S, et al. External validation of the hospital frailty risk score in France. Age Ageing. 2022;51(1):afab126. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Appendix S1
Data Availability Statement
The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions.