External validation of the Hospital Frailty Risk Score in France

Abstract

Background

The Hospital Frailty Risk Score (HFRS) has made it possible internationally to identify subgroups of patients with characteristics of frailty from routinely collected hospital data.

Objective

To externally validate the HFRS in France.

Design

A retrospective analysis of the French medical information database.

Setting

743 hospitals in Metropolitan France.

Subjects

All patients aged 75 years or older hospitalised as an emergency in 2017 (n = 1,042,234).

Methods

The HFRS was calculated for each patient based on the index stay and hospitalisations over the preceding 2 years. Main outcome measures were 30-day in-patient mortality, length of stay (LOS) >10 days and 30-day readmissions. Mixed logistic regression models were used to investigate the association between outcomes and HFRS score.

Results

Patients with high HFRS risk were associated with increased risk of mortality and prolonged LOS (adjusted odds ratio [aOR] = 1.38 [1.35–1.42] and 3.27 [3.22–3.32], c-statistics = 0.676 and 0.684, respectively), while it appeared less predictive of readmissions (aOR = 1.00 [0.98–1.02], c-statistic = 0.600). Model calibration was excellent. Restricting the score to data prior to index admission reduced discrimination of HFRS substantially.

Conclusions

HFRS can be used in France to determine risks of 30-day in-patient mortality and prolonged LOS, but not 30-day readmissions. Trial registration: Reference ID on clinicaltrials.gov: ID: NCT03905629.

Key Points

• The HFRS predicts mortality and extended length of stay risks among French hospitalised older adults, but not readmissions.

• It is mainly applicable for service planning, but not sufficient for individual clinical decision-making.

• The HFRS has the potential be used as a frailty risk marker in epidemiological studies using administrative data worldwide.

Introduction

Hospitals are increasingly faced with older patients with many comorbidities, requiring multi-professional assessment and management [1–3]. In France in 2017, over half (56%) of patients aged 75 years and over presenting to emergency departments (EDs) were admitted for acute hospital care [4]. Frail older people admitted to hospital have increased risk of adverse events, have long hospital stays and high rates of long-term care use [1, 5–7]. Frailty can generally be defined as a state of increased vulnerability of older patients towards adverse outcomes when exposed to a stressor event [8]. Given the complex nature of the frailty construct, no existing frailty tool has emerged as a gold standard method to predict individual patient outcomes [9, 10].

Multiple tools have been developed to identify frailty in both clinical and research settings, typically requiring face-to-face assessment by trained observers, some requiring specific clinical measurements, such as hand-grip strength [10]. However, the implementation of such tools remains difficult in busy urgent care settings, in which screening can be achieved in 52% of patients at best [11]. Thus, there is widespread interest in using electronic health records and administrative datasets to mine Big Data, derive frailty prediction models and help direct holistic care to patients most likely to benefit [12–14]. Other key applications include service planning, benchmarking, evaluating health policies and developing markers for epidemiological research. Frailty, as a multi-dimensional construct, is not routinely documented in electronic health records and has not yet been captured by a single diagnosis code [15]. Accumulated deficits and high resource use are commonly recognised by clinicians as key indicators of frailty in acute care settings [16].

The Hospital Frailty Risk Score (HFRS) [17] was recently developed and validated in England using national hospital episode statistics data, building on the results of a preliminary cluster analysis to identify a subgroup of older patients displaying characteristics of frailty (of whom 48% had died over the 2-year follow-up period). Intermediate and high levels of frailty were significantly associated with increased risks of 30-day mortality, prolonged hospital length of stay (LOS) and 30-day readmission [17]. A key strength of this score is that it has the potential to be implemented in administrative datasets using International Classification of Diseases, 10th revision (ICD-10) codes internationally. However, given the risk of potential discrepancies in terms of routine coding practices, this requires prior validation to confirm that the model achieves satisfactory discrimination and calibration in the specific context of application [18, 19].

The main objective of this study was to determine if HFRS can be used in French hospitals to discriminate outcomes in patients aged 75 years and older admitted to acute care, with satisfactory prediction accuracy across settings.

Methods

Study design and participants

We conducted a retrospective analysis of routinely collected secondary care data from the French nationwide Medical Information System database (Programme de Médicalisation des Systèmes d’Information [PMSI], source: ATIH). Inclusion criteria were replicated from the initial HFRS validation study [17]. We included all patients aged 75 years and older (on the day of the index admission) hospitalised as an emergency in Metropolitan France over a period of 1 year between 1 January and 31 December 2017. In case of multiple admissions of the same patient, only the first (index) admission was used for the analysis, in order to maintain population homogeneity. Considering their low frequency, patients with missing data for city of residence median income or medical accessibility were also excluded from the study population as described in the flowchart (Figure 1). The predictive ability of the HFRS was estimated on three binary outcomes: 30-day in-patient mortality, LOS > 10 days and 30-day emergency readmissions. Odds ratios (ORs) and c-statistics were calculated and compared to the original English results in order to validate the use of the HFRS in France.

Figure 1 .

Flowchart.

Data source and outcomes

We used anonymised data from the French PMSI database, which contains routinely collected data from all public and private hospitals in France. Each in-patient stay is allocated to a single Diagnosis-Related Group based on standard discharge abstracts containing compulsory information about the patient, primary and secondary diagnoses using the ICD-10 codes [20].

The main outcome was in-hospital mortality within 30 days from the beginning of the index admission. Other outcomes included long LOS (>10 days) and emergency readmissions up to 30 days from discharge from the index admission (excluding patients who had died in hospital).

Patient covariates considered included age, sex, Charlson comorbidity index categories (0, 1, 2 or 3+, as previously described in Gilbert et al.) [17, 21, 22] and admission history (i.e. number of hospital bed-days during the past 2 years from the index admission as a continuous variable and, for descriptive purposes only, the number of previous admissions in this period). Patients’ socioeconomic status (median household income in the city of residence, continuous) and medical accessibility (mean number of family medicine consultation/year/inhabitant in the city of residence, continuous) were associated with patients’ city of residence postcode and provided by the National Institute of Statistics and Economic Studies and Department of Research, Studies, Evaluation and Statistics respectively. Those two variables were not available for 0.8% of patients that were consequently excluded.

The hospital status (university hospital, regional or local hospital, private clinic) was retrieved for each index hospitalisation, the aim being to identify and describe possible specificities in terms of case-mix and level of frailty according to different types of settings.

Hospital Frailty Risk Score

The HFRS was calculated for each included patient [17], based on the ICD-10 diagnoses documented in their index emergency admission and hospital records and retrieved from anonymised discharge summaries going back 2 years. The ICD-10 codes considered corresponded to the list used to create the HFRS in England [17]. These included principal as well as secondary diagnoses, in the same way restricted to the third figure (for example ‘R26’, and not ‘R26.2’). Patients were allocated to one of three categories based on their HFRS score, applying the same three thresholds used in the original study: low risk <5, intermediate risk 5–15 and high risk >15.

Statistical analysis

Patient characteristics were described using mean and standard deviation for continuous variables, frequencies and percentages for qualitative variables.

To estimate the association of the HFRS categories with outcomes, we fitted mixed logistic regression models with random effects to capture hospital variation. Models were first estimated without adjustment and then with adjustment on age, sex, socioeconomic data, admission history and the Charlson comorbidity index categories. Associations between HFRS categories and each outcome were evaluated with ORs, and discrimination with C-statistics, accompanied by their 95% confidence intervals. Model calibration was assessed by plotting the observed frequency of events per tenths of predicted risks [23]. Models assessing readmissions excluded patients who died during the index admission.

We considered diagnostic information from the patient’s index admission and those occurring in the previous 2 years to build ‘standard’ HFRS. As sensitivity analysis we excluded information from the index admission to build a ‘historic’ score.

We performed two other sensitivity analyses. First, we considered HFRS as a continuous variable using splines and functional forms to model its association with outcomes. Then, we considered a survival approach using Cox’s proportional hazards model to estimate hazard ratios (HRs) for mortality and readmissions. To take into account the influence of the competing risk of in-patient mortality on the association of HFRS with time to readmission, we also ran a proportional hazards model for the sub-distribution of a competing risk (i.e. the ‘Fine-Gray’ model [24]) with robust variance estimator to account for clustering within hospitals.

Data manipulation and analyses were performed using SAS software (version 9.4; SAS Institute Inc., Cary, NC).

Results

A total of 1,042,234 patients aged 75 years and older hospitalised as an emergency in 743 hospitals were included (Figure 1). Patient characteristics are presented in Table 1. Patients with higher risks of frailty were generally older and more likely to be female. Number of past admissions was also greater among these patients. Finally, university and public hospitals cared for patients with higher risks of frailty than private clinics. Comorbidity scores and outcomes are presented in Table 2.

Table 1 .

Characteristics of patients by HFRS category—mean (SD) and frequencies (%)

		HFRS
		Overall	<5	5–15	>15
Variable	Value	N = 1,042,234	N = 472,816 (45.4%)	N = 386,894 (37.1%)	N = 182,524 (17.5%)
Male, N (%)		417,487 (40.1%)	200,594 (42.4%)	149,736 (38.7%)	67,157 (36.8%)
Age, mean (SD)		84.9 (5.8)	83.9 (5.8)	85.5 (5.8)	86.2 (5.6)
Living in low-income areas (5th quintile of the city of residence median incomea)		208,703 (20.0%)	92,508 (19.6%)	78,306 (20.2%)	37,889 (20.8%)
Medical accessibility, mean (SD)		4.1 (1.1)	4.1 (1.1)	4.1 (1.1)	4.1 (1.2)
Number of past hospital bed-days, mean (SD)		10.7 (18.2)	3.9 (7.8)	11.4 (16.4)	26.9 (28.0)
Nb past admissions (including ambulatory), mean (SD)		1.6 (2.2)	0.9 (1.6)	1.7 (2.2)	3.1 (2.9)
Nb past admissions (including ambulatory), N (%)	0	411,438 (39.5%)	253,594 (53.6%)	132,666 (34.3%)	25,178 (13.8%)
	1	238,835 (22.9%)	106,960 (22.6%)	96,169 (24.9%)	35,706 (19.6%)
	2	157,070 (15.1%)	59,028 (12.5%)	63,626 (16.4%)	34,416 (18.9%)
	≥3	234,891 (22.5%)	53,234 (11.3%)	94,433 (24.4%)	87,224 (47.8%)
Index stay duration (nights), mean (SD)		8.6 (9.1)	6.2 (6.7)	10.0 (9.3)	11.9 (11.8)
Hospital status, N (%)	University Hospital	202,632 (19.4%)	89,631 (19.0%)	74,383 (19.2%)	38,618 (21.2%)
	Private	97,140 (9.3%)	59,025 (12.5%)	28,636 (7.4%)	9,479 (5.2%)
	Public	742,462 (71.2%)	324,160 (68.6%)	283,875 (73.4%)	134,427 (73.6%)
Charlson, mean (SD)		1.5 (1.4)	0.9 (1.1)	1.6 (1.4)	2.5 (1.6)

^aPatients’ city of residence median income was considered continuously in the final models.

Table 2 .

Patients’ outcome by HFRS category—mean (SD) and frequencies (%)

		HFRS	Low risk	Intermediate risk	High risk
		Overall	<5	5–15	>15
Variable		N = 1,042,234	N = 472,816 (45.4%)	N = 386,894 (37.1%)	N = 182,524 (17.5%)
Mortality within 30 days, N (%)		74,929 (7.2%)	23,160 (4.9%)	32,743 (8.5%)	19,026 (10.4%)
LOS > 10 days, N (%)		304,303 (29.2%)	85,189 (18.0%)	139,161 (36.0%)	79,953 (43.8%)
Emergency readmissions within 30 days, N (%)		95,171 (9.1%)	38,536 (8.2%)	36,722 (9.5%)	19,913 (10.9%)

In-patient 30-day mortality was significantly higher in intermediate and high-risk HFRS groups compared to low risk (Table 3, HFRS crude OR = 1.79 [1.76–1.82]/2.29 [2.25–2.34], c-statistic = 0.617). Likewise, there was an association of increased levels of HFRS with prolonged hospital LOS (OR = 2.56 [2.54–2.59]/3.59 [3.55–3.64], c-statistic = 0.669). These trends were similar after adjustment on covariates (adjusted ORs = 1.34 [1.32–1.37]/1.38 [1.35–1.42] for mortality and 2.34 [2.32–2.37]/3.27 [3.22–3.32] for prolonged LOS, with c-statistics = 0.676 and 0.684 respectively). Even though higher HFRS levels were associated with increased risks of readmission in the unadjusted model (OR = 1.24 [1.22–1.26]/1.48 [1.45–1.51], c-statistic = 0.565), we found no clear association after adjustment on covariates, and discrimination for this outcome was poor (Table 3, adjusted OR = 1.04 [1.02–1.05]/1.00 [0.98–1.02], c-statistic = 0.600). Model calibration was excellent, with a calibration slope around 1 for all models and a calibration line close to the diagonal line (Figure 2).

Table 3 .

Relationship between HFRS frailty risk category and outcomes (n = 1,042,234)

Outcome, HFRS frailty risk	%	Crude OR (95% CI)	Adjusted OR (95% CI)
30-day in-patient mortality
Low risk (<5)	4.9%	1.00	1.00
Intermediate risk (5–15)	8.5%	1.79 (1.76–1.82)	1.34 (1.32–1.37)
High risk (>15)	10.4%	2.29 (2.25–2.34)	1.38 (1.35–1.42)
C-statistic		0.617 (0.615–0.619)	0.676 (0.674–0.678)
LOS > 10 days
Low risk (<5)	18.0%	1.00	1.00
Intermediate risk (5–15)	36.0%	2.56 (2.54–2.59)	2.34 (2.32–2.37)
High risk (>15)	43.8%	3.59 (3.55–3.64)	3.27 (3.22–3.32)
C-statistic		0.669 (0.668–0.670)	0.684 (0.683–0.685)
Emergency readmission within 30 daysa
Low risk (<5)	8.2%	1.00	1.00
Intermediate risk (5–15)	9.5%	1.24 (1.22–1.26)	1.04 (1.02–1.05)
High risk (>15)	10.9%	1.48 (1.45–1.51)	1.00 (0.98–1.02)
C-statistic		0.565 (0.563–0.567)	0.600 (0.598–0.602)

Models were fitted with random effects to capture hospital variation, first without any adjustment and then adjusted for age, gender, hospital bed-days, primary care access, city median income and Charlson’s index categories. Results displayed are ORs with their 95% CIs. CI, confidence interval.

^aDeaths censored for readmissions: 68,875.

Figure 2 .

Calibration assessment for mixed logistic regression models. Crude (A, C, E) and adjusted models (B, D, F) for 30 day in-patient mortality, prolonged LOS > 10 days and 30-day readmissions, respectively.

Using only data prior to the index admission reduced predictive abilities of the HFRS for all outcomes significantly: adjusted OR for in-patient 30-day mortality were 1.02 (1.00–1.05) for intermediate risk and 1.05 (1.01–1.08) for high risk (Appendix 1 is available in Age and Ageing online), with lower discriminations (c-statistics = 0.648 for ‘historic’ HFRS versus 0.676 for ‘standard’ HFRS). Furthermore, we observed decreased odds of prolonged LOS for high-risk patients on the ‘historic’ HFRS (Appendix 1 is available in Age and Ageing online, adjusted OR = 0.97 [0.95–0.99], c-statistic = 0.603).

Substituting the HFRS categories by using the HFRS score as a continuous variable did not improve model discrimination (Appendix 2 is available in Age and Ageing online). Using a survival approach to assess in-hospital mortality and 30-day re-admission (Appendix 3 is available in Age and Ageing online), Cox’s HRs were consistent with the results of logistic regression analyses. Furthermore, considering the competing risk of mortality did not affect the results for readmissions (Appendix 3 is available in Age and Ageing online).

Discussion

Principal findings

Our study confirms that HFRS can be used in France to predict 30-day in-patient mortality and prolonged LOS, but not 30-day readmissions. Patients displayed similar characteristics to the English validation cohort, with comparable distributions of low, intermediate and high risk of frailty and similar numbers of past admissions. The thresholds of 5, 5–15 and >15 used to define these categories seemed adequate, and there was no added value in constructing the score as a continuous variable. Model calibration was excellent for all outcomes.

Another key finding of our study was that the ‘historic’ HFRS score (i.e. suppressing data from the index admission) performed no better than the model with adjustment factors alone to predict these outcomes. This compares to the original validation cohort, in which the ‘historic’ HFRS also had weaker amplitude of effect and discrimination (30-day mortality adjusted OR = 1.31 (1.28–1.34) for intermediate and 1.35 (1.31–1.39) for high risk; c-statistic = 0.59) [17], and suggests that the HFRS would be of limited use to direct clinical care of older patients upon arrival to EDs but could still be useful as a frailty mapping tool at the system level.

Comparison with other studies

Since its publication in May 2018, the use of HFRS has been successfully validated in various other countries ([25–34], Appendix 4 is available in Age and Ageing online). Our results were similar to previous studies for mortality prediction and LOS, with similar c-statistics (0.55–0.76 for in-patient mortality, 0.64–0.73 for prolonged LOS). The discrimination of our adjusted model for these two outcomes (both c-statistics around 0.68) was just below the threshold of 0.70 defining ‘good’ discrimination but remains insufficient for directly guiding clinical care. By way of comparison, the HFRS seemed to achieve good discrimination (i.e. 0.70 or above) for these outcomes in specific contexts such as acute heart failure or myocardial infarction [29, 30]. However, results for unplanned readmission risk are much less consistent across studies (Appendix 4 is available in Age and Ageing online).

Patients with higher levels of HFRS tended to have longer LOS in France compared to England. This could highlight differences in the way health systems and community services are organised in the different countries, for example it might be that England’s focus on rapid discharge might be associated with increased readmissions [35, 36].

Strengths and limitations of this study

This study was based on a very large population of patients admitted in any public or private hospital in metropolitan France, for a whole year, taking into account hospital variation and seasonality. The population is representative of all older subjects admitted to EDs in France or other comparable countries. Both discrimination and calibration were assessed in the validation process, and results were robust to several sensitivity analyses. This brings a good level of confidence in the applicability of the score to all French hospitals. However, a few limitations need to be highlighted. First, due to issues regarding authorisation to access and linking of database to national mortality registries, only in-patient mortality data were directly available from the PMSI database, which may impact validity and generalisability of our results. In particular, the model built to predict readmissions could only exclude patients who had died in hospital, and new nursing home admissions were not captured in our study [37]. Second, as all proxy measures based on collected diagnoses, the HFRS score and results of our study might be subject to coding bias. In addition, patients with more frequent admissions and longer hospital stays may have, consequently, more coded information and might be more likely to be labelled as ‘frail’. More generally, hospital administrative data are not usually collected for research purposes, and diagnosis for instance can be inaccurately coded, leading to missing data bias. We cannot rule out the possibility that other residual confounders may have influenced our findings, owing to potential inaccuracies inherent to these kinds of data.

Finally, one of the main limitations of HFRS is that the score does not take into account the severity of the presenting condition [38–41]. A previous study has highlighted the limited value of HFRS for risk prediction in intensive care units, where prognosis is essentially guided by the severity of the acute condition [27]. Further iterations of the score should aim to take this into account, and we will also follow-up with great interest the results of the HAVEN risk score construct (Hospital alerting via electronic noticeboard), which will compile ‘static’ variables such as age, HFRS and comorbidity indices, with a set of ‘dynamic’ variables reflecting disease acuity [42].

Conclusions and implications

This study confirms the validity of HFRS for use on French national data in relation to mortality and prolonged LOS. This is consistent with the results observed in other countries and paves the way for the conduct of international studies using equivalent datasets in populations of older subjects admitted from EDs. The HFRS is mainly applicable as a marker of frailty risk for service planning and epidemiological studies using hospital data, but in keeping with other frailty scores, not suitable for individual patient decision-making.

Acknowledgements

The authors would like to thank Léa Pascal for her contribution to this work.

Declaration of Conflicts of Interest

None.

Declaration of Sources of Funding

This study was supported by a grant from Hospices Civils de Lyon (direction de la recherche clinique et de l’innovation), France. The funding source had no influence in the design, data collection, interpretation of data, writing of the report or decision to submit the article for publication.