Predicting 30-day mortality after hip fracture surgery

Abstract

Objectives

The National Hip Fracture Database (NHFD) publishes hospital-level risk-adjusted mortality rates following hip fracture surgery in England, Wales and Northern Ireland. The performance of the risk model used by the NHFD was compared with the widely-used Nottingham Hip Fracture Score.

Methods

Data from 94 hospitals on patients aged 60 to 110 who had hip fracture surgery between May 2013 and July 2013 were analysed. Data were linked to the Office for National Statistics (ONS) death register to calculate the 30-day mortality rate. Risk of death was predicted for each patient using the NHFD and Nottingham models in a development dataset using logistic regression to define the models’ coefficients. This was followed by testing the performance of these refined models in a second validation dataset.

Results

The 30-day mortality rate was 5.36% in the validation dataset (n = 3861), slightly lower than the 6.40% in the development dataset (n = 4044). The NHFD and Nottingham models showed a slightly lower discrimination in the validation dataset compared with the development dataset, but both still displayed moderate discriminative power (c-statistic for NHFD = 0.71, 95% confidence interval (CI) 0.67 to 0.74; Nottingham model = 0.70, 95% CI 0.68 to 0.75). Both models defined similar ranges of predicted mortality risk (1% to 18%) in assessment of calibration.

Conclusions

Both models have limitations in predicting mortality for individual patients after hip fracture surgery, but the NHFD risk adjustment model performed as well as the widely-used Nottingham prognostic tool and is therefore a reasonable alternative for risk adjustment in the United Kingdom hip fracture population.

Cite this article: Bone Joint Res 2017;6:550–556

Article focus

• Numerous models, both generic and hip fracture-specific, have been developed to predict mortality after hip fracture surgery.

• The Nottingham Hip Fracture Score has been validated and applied widely in the United Kingdom for predicting mortality in hip fracture patients.

• The refined risk adjustment model used by the National Hip Fracture Database was compared with the Nottingham Hip Fracture Score using national clinical audit data.

Key messages

• The model used by the National Hip Fracture Database performed as well as the Nottingham Hip Fracture Score.

• The National Hip Fracture Database model is a suitable alternative to the Nottingham Hip Fracture Score for risk adjustment in the United Kingdom hip fracture population.

Strengths and limitations

• Patients who had any type of surgery for hip fracture were included in the study, and thus the findings are applicable to the entire hip fracture population.

• Data quality in this study was better than in earlier validation studies of the Nottingham model, and missing data were managed using the robust approach of multiple imputation.

• There may be residual confounding due to risk factors that were not, or were only partly, captured in the dataset.

Introduction

Over 65 000 people aged 60 years or older suffer a hip fracture in England, Wales and Northern Ireland every year.¹ The injury is associated with increased risk of death, with only approximately 70% of patients surviving one year after their fracture, and 7% of patients dying within 30 days of admission (4622/64 858).¹ Numerous patient factors are associated with mortality after hip fracture. It is important that such case-mix variation is considered in the analysis and interpretation of hospital-level results in national clinical audits, not least when this information is used to benchmark services.

The National Hip Fracture Database (NHFD) has been publishing information on patient outcomes following hip fracture in England, Wales and Northern Ireland since 2007. Hospital figures for 30-day mortality rates have always been risk-adjusted by the NHFD, with periodic review of evidence on prognostic models for hip fracture patients to support improvement of its risk adjustment method. This latest refinement to the NHFD risk adjustment model was first implemented in 2014.² The NHFD model features the same six patient factors as the model developed by Holt et al³ from the Scottish Hip Fracture Audit, although different categories are used for some variables.

Prognostic tools specific to patients with hip fracture are unsurprisingly more reliable than generic models,^4,5 but their performances are affected by characteristics of the populations from which they were derived. Different models incorporate a variety of patient characteristics,^3-7 but it is not always clear whether the additional burden of data collection for more complex models is justified by enhanced predictive performance.

The Nottingham Hip Fracture Score is one of the most frequently used and extensively validated outcome prediction models for hip fracture patients.^6-8 It was developed in an English hospital setting and, in 2014, over half (51.7%) of hospitals in England, Wales and Northern Ireland (93/180) reported occasional or routine use of this tool.⁹ It has been validated in the United Kingdom and elsewhere against generic models for post-operative mortality such as the Surgical Outcome Risk Tool (SORT) and hip-specific models such as the Almelo Hip Fracture Score (AHFS).^6,8,10-12 It has been recalibrated since its initial development and the creators have recommended further adjustments to account for changes in the hip fracture population.⁸ In this study, the risk model used by the NHFD for 30-day mortality^2,9,13 was evaluated by comparing its performance against the Nottingham model.

Patients and Methods

Data sources

The study used data collected by the NHFD as part of the Royal College of Physicians and Association of Anaesthetists of Great Britain and Ireland collaborative ‘sprint’ audit on anaesthetic practice. The Anaesthetic Sprint Audit of Practice (ASAP) ran in parallel with standard NHFD data collection over three months in 2013 at 95 hospitals in England, Wales and Northern Ireland.¹⁴ During this time, hospitals submitted data on additional audit fields beyond the standard NHFD dataset, which enabled the estimation of individual patients’ risk of 30-day mortality using both NHFD and Nottingham models.

Ethical approval was not required since the NHFD has Section 251 approval from the Health Research Authority’s Confidentiality Advisory Group to collect details of hip fracture patients and link their data with the Office for National Statistics (ONS) death register.

Patient population

The study included patients who were admitted for hip fracture, between 1 May 2013 and 31 July 2013, to a NHS hospital in England, Wales or Northern Ireland that participated in ASAP. Patients with known operation type were selected for analysis if they were aged between 60 and 110 years, so long as their mortality status at 30 days after surgery could be confirmed by linking their record with the ONS death register.

The risk models

The NHFD model contains six variables:² age; gender; American Society of Anesthesiologists (ASA) physical status grade; ability to walk indoors; fracture type; and source of admission. The Nottingham model contains seven variables: age; gender; number of comorbidities; abbreviated mental test score on admission; haemoglobin concentration; living in an institution; and malignant disease. The Nottingham model was developed with age stratified into three categories (< 66, 66 to 85, and ⩾ 86 years). This study however used the slightly different categories that were applied in the ASAP study.⁷ These corresponded to the categories used in the NHFD model. There was also some overlap between the two models in the other patient risk factors that they contain (Table I).

Table I.

Patient factors in the Nottingham and National Hip Fracture Database (NHFD) models

Patient factor	In NHFD	NHFD categories	In Nottingham model	Nottingham categories
Age (yrs)	Yes		Yes
		60 to 69		60 to 69
		70 to 79		70 to 79
		80 to 89		80 to 89
		90 to 110		90 to 110
Gender	Yes		Yes
		Male		Male
		Female		Female
Source of admission	Yes		N/A
		Own home/sheltered housing
		Not from own home
ASA grade	Yes		N/A
		1 or 2
		3
		4 or 5
Walking indoors ability	Yes		N/A
		Regularly walked without aids
		Regularly walked with one aid, two aids or frame
		Wheelchair or bedbound
Fracture type	Yes		N/A
		Intracapsular
		Extracapsular (or other)
Number of comorbidities	N/A		Yes
				0 to 1
				⩾ 2
AMTS on admission	N/A		Yes
				Number from 0 to 10
Hb on admission	N/A		Yes
				Numeric value
Living in an institution	N/A		Yes
				Yes
				No
Malignant disease	N/A		Yes
				Yes
				No

ASA, American Society of Anesthesiologists; AMTS, Abbreviated Mental Test Score; Hb, Haemoglobin concentration; NHFD, National Hip Fracture Database; N/A, Not Applicable

Statistical analysis

We used a standard approach for assessing and improving prediction models¹⁵ whereby a model is refined in one dataset before its performance is examined in a different group of patients.

We used a ‘development’ dataset of patients admitted between 1 May 2013 and 15 June 2013 to recalculate (recalibrate) the coefficients of the risk factors in the NHFD and Nottingham models. Each patient’s probability of death within 30 days of surgery for hip fracture could then be predicted from the regression coefficients of the two models.^16,17

The resulting risk equations for the two models were applied to the ‘validation’ dataset of patients admitted between 16 June 2013 and 31 July 2013 to examine their predicted risk of death.^16,17

The ‘discrimination’ of a model describes its ability to differentiate between patients who survived or died using the area under the receiver-operating characteristic (ROC) curve. We interpreted values of this ‘c-statistic’ as indicating poor model performance if they were below 0.70, as moderate for 0.70 to 0.79, and as good performance for 0.80 to 0.89.^12,18

The ‘calibration’ of a model compares the predicted and observed mortality rates between groups of patients who are grouped based on their predicted mortality risk. We assessed calibration visually and with the Hosmer-Lemeshow test, with patients allocated to eight predicted risk groups. Goodness of fit was considered adequate if the p-value was less than 0.05.¹⁹

We addressed missing data using multiple imputation by chained equations (MICE), that is, missing data assumed to be missing at random. Rubin’s rules were applied to produce 20 imputed datasets.²⁰ All analyses were carried out using STATA version 14.1 (StataCorp LP, College Station, Texas).

Results

During the study period, 8290 patients with hip fracture presented to hospitals that participated in ASAP. The following exclusions were made: n = 116 missing the date of operation; n = 96 missing the type of operation; n = 172 where survival after operation was unknown; and n = 1 with missing data for six risk factors. After exclusions, 7905 patients from 94 hospitals were included in the analysis (Fig. 1).

Fig. 1.

Flow diagram of study participation. ASAP, Anaesthesia Sprint Audit of Practice.

The distribution of patient factors was relatively similar in the development and validation datasets (Table II). The overall mortality rate was 5.89% (466/7905). The mortality rate within 30 days of surgery for hip fracture was 6.40% in the development dataset (259/4044), but was slightly lower at 5.36% in the validation dataset (207/3861).

Table II.

Patient factors and 30-day mortality rate in development and validation datasets

	Full dataset			Development set, n = 4044			Validation set, n = 3861
	All patients		Died	All patients		Died	All patients		Died
Patient factor	n	%	%	n	%	%	n	%	%
Age (yrs)
60 to 69	711	9.0	3.1	373	9.2	2.9	338	8.8	3.3
70 to 79	1780	22.5	4.0	871	21.5	4.8	909	23.5	3.2
80 to 89	3673	46.5	5.6	1906	47.1	5.9	1767	45.8	5.3
90 to 110	1741	22.0	9.6	894	22.1	10.5	847	21.9	8.6
Gender
Female	5719	72.3	4.6	2940	72.7	4.7	2779	72.0	4.5
Male	2186	27.7	9.3	1104	27.3	10.9	1082	28.0	7.7
Source of admission
Own home/sheltered housing	5996	75.9	4.9	3065	75.8	5.3	2931	75.9	4.4
Not from own home	1909	24.1	9.1	979	24.2	9.8	930	24.1	8.4
ASA grade
1 or 2	2482	31.4	1.8	1246	30.8	1.8	1236	32.0	1.8
3	4264	53.9	6.5	2197	54.3	7.0	2067	53.5	6.0
4 or 5	954	12.1	13.8	488	12.1	15.6	466	12.1	12.0
Unknown/missing	205	2.6	5.4	113	2.8	6.2	89	2.3	4.3
Walking indoors ability
Without aids	3784	47.9	4.0	1926	47.6	4	1858	48.1	4.0
One aid, two aids or frame, wheelchair or bedbound	4003	50.6	7.5	2055	50.8	8.4	1948	50.5	6.5
Unknown/missing	118	1.5	11.9	63	1.6	12.7	55	1.4	10.9
Fracture type
Intracapsular	4563	57.7	6.1	2315	57.2	6.5)	2248	58.2	5.7
Extracapsular, including other	3342	42.3	5.6	1729	42.8	6.3	1613	41.8	4.8
Comorbidities
0 to 1	3495	44.2	3.8	1807	44.7	3.7	1688	43.7	4.0
⩾ 2	3222	40.8	7.7	1744	43.1	8.3	1478	38.3	6.9
Missing	1188	15.0	7.2	493	12.2	9.5	695	18.0	5.5
AMTS on admission
0 to 6	2330	29.5	8.9	1219	30.1	9.4	1111	28.8	8.5
7 to 10	5047	63.8	4.0	2552	63.1	4.6	2495	64.6	3.4
Unknown/missing	528	6.7	10.4	273	6.8	5.3	221	5.7	11.3
Hb on admission
> 10 g/dl-1	7354	93	5.5	3732	92.3	5.9	3622	93.8	5.2
⩽ 10 g/dl-1	551	7.0	10.7	312	7.7	12.8	239	6.2	7.9
Living in an institution
No	5996	75.9	4.9	3065	75.8	5.3	2931	75.9	4.4
Yes	1909	24.1	9.1	979	24.2	9.8	930	24.1	8.4
Malignant disease
No	7134	90.2	5.4	3638	90	5.9	3496	90.5	5.0
Yes	771	9.8	10.2	406	10	11.3	365	9.5	9.0

ASA, American Society of Anesthesiologists; AMTS, Abbreviated mental test score; Hb, Haemoglobin concentration

Model recalibration and validation

The development dataset was used to derive coefficients for the NHFD and Nottingham models and produce the following risk scores:

NHFD risk score = exp (- 4.723 + (0.435 x age 70 to 79) + (0.518 x age 80 to 89) + (1.060 x age 90 to 110) + (0.894 x male) + (-0.275 x not admitted from home) + (1.164 x ASA 3) + (1.933 x ASA 4 or 5) + (0.358 x walk with at least one aid) + (-0.055 x extracapsular fracture)).

Nottingham risk score = exp (- 3.955 + (- 0.407 x age 70 to 79) + (0.485 x age 80 to 89) + (1.083 x age 90 to 110) + (0.883 x male) + (0.558 x Hb ⩽ 10 g dl⁻¹) + (0.662 x cancer) + (- 0.537 x AMTS ⩾ 7) + (0.263 x living in institution) + (0.755 x ⩾ 2 comorbidities)).

The NHFD and Nottingham models displayed similar levels of performance (Table III). Both were moderately able to distinguish between patients at high and low risk of death within 30 days, with a c-statistic of 0.71 for the NHFD model and 0.70 for the Nottingham model in the validation dataset.

Table III.

Performance of Nottingham and NHFD models

Dataset	Model	Patients, n	Hosmer-Lemeshow test			Area under ROC curve
			chi-squared	df	p-value	C-statistic	95% CI
Development	Nottingham	4044	6.27	6	0.394	0.73	0.70 to 0.76
	NHFD	4044	11.3	5	0.078	0.74	0.71 to 0.77
Validation	Nottingham	3861	6.10	6	0.599	0.70	0.67 to 0.74
	NHFD	3861	14.9	5	0.029	0.71	0.68 to 0.75

CI, confidence intervals; df, degrees of freedom; NHFD, National Hip Fracture Database; Nottingham, Nottingham Hip Fracture Score; ROC, receiver operating characteristic

Model calibration was similar in both models, with a slightly reduced range of predicted risk in the validation dataset (Fig. 2). The Nottingham model displayed a better fit (p-value = 0.599 for Hosmer-Lemeshow test) than the NHFD model (p-value = 0.029), but both were inconsistent in their estimations of mortality risk for patients in the eight risk groups (Fig. 2).

Observed versus predicted 30-day mortality by risk group, Nottingham and National Hip Fracture Database (NHFD) models. Figure 2a) shows the development dataset, while b) shows the validation dataset.

Discussion

Main findings

Case-mix adjustment is important in the development and refinement of prediction models, as potential confounding caused by differences in the populations of individual hospitals must be addressed. The risk model used by the NHFD to adjust 30-day mortality rates for individual hospitals features the same six patient factors as previous NHFD risk models, but with different categories for some variables, and is based on a regression equation rather than a classification tree.²¹

In this evaluation of the NHFD risk model, its performance was comparable with one of the most commonly used and well validated outcome prediction tools for hip fracture – the Nottingham Hip Fracture Score. Both models displayed moderate discriminative power, with the NHFD model achieving a slightly higher c-statistic of 0.71 in the validation dataset. The two models also displayed moderate levels of calibration, although there was some inconsistent estimation of mortality risk for patients in different risk groups when applied to the validation dataset. These findings are consistent with studies that have found limited goodness of fit for the Nottingham model and the model described by Holt et al.^5,10

Strengths and limitations

Previous studies have excluded certain patient groups, such as those who underwent total hip arthroplasty,⁵ but this study included patients who had any type of surgery for hip fracture. The findings could therefore be applied to the whole hip fracture population. The study has several other strengths. It used better quality data than earlier validation studies of the Nottingham model because a high level of data completeness was achieved in the ASAP, where hospitals were excluded if their case ascertainment was 80% or less. We adopted the more robust approach of multiple imputation to manage missing data,²² while other validation studies limited themselves to complete case analysis by excluding records with missing data.^6,10

Validation of the Nottingham model using the original and subsequently recalibrated risk score was not feasible as the regression coefficients used to calculate the current Nottingham Hip Fracture Score are not publicly available.⁸ Comorbidities were only recorded in the ASAP dataset in terms of whether they were present or not, so it was not possible to determine whether patients without specific recorded comorbidities truly did not have those conditions.

Finally, both models include a selection of patient characteristics associated with death following surgery for hip fracture,²³ but they do not capture all potential factors of interest. Future refinement of either model should be guided by the availability and reliability of recorded variables.

Implications for further research and clinical practice

This study has demonstrated that the NHFD model performs comparably with the Nottingham model, and meets current standards for hip fracture outcome prediction in the United Kingdom hip fracture population. Therefore, the NHFD model could be considered for use as an alternative to the Nottingham model, especially if variables (such as haemoglobin concentration or the nature and number of comorbidities) are poorly recorded or missing.

There is room for improvement in both models. When used to calculate risk-adjusted outcomes of hospitals, there is a small chance of wrongly classifying a hospital as an outlier. However, compared with making predictions for individual patients, the size of the error will be small. This is because the risk adjustment process aims to estimate the average risk of all patients treated at different hospitals, and the differences in the average risks among hospitals is much less than the differences in the risks among individual patients.

If these models are to be used to predict mortality at the individual patient level, we suggest that they be further refined and validated. For example, regular recalibration of both models is appropriate given the trend in decreasing 30-day mortality rate over recent years (7.1% in 2015 compared with 8.0% in 2013).^1,13 Similarly, changes to the profile of hip fracture patients will need to be taken into account in future updates to the models.²⁴ However, the risk of death faced by an individual patient cannot be fully defined by any model based on risk factors present on admission, since it will obviously also depend on the quality of care they subsequently receive. The importance of prompt surgery and peri-operative orthogeriatric assessment are just two examples of aspects of care that impact on patients’ mortality risk.

While mortality is the most commonly measured patient outcome, there are many other outcomes relevant to the assessment of care delivered to hip fracture patients. For example, functional independence and quality of life in the intermediate future following hip fracture are of interest and importance to patients and their families, care providers and policy makers. Future studies would benefit from assessing the performances of these tools in their ability to predict differences among patients on these outcomes. This would lead to a better understanding of how different patient groups (such as those at relatively high risk or low risk of death within 30 days of surgery) are affected by hip fracture in the months after completing hospital treatment.

Funding Statement

Carmen Tsang reports that funding has been received from Healthcare Quality Improvement Partnership (HQIP) which is related to this paper.

The NHFD is commissioned by the Healthcare Quality Improvement Partnership (HQIP), and managed by the Clinical Effectiveness and Evaluation Unit of the Royal College of Physicians (RCP) as part of the Falls and Fragility Fracture Audit Programme (FFFAP).

The Anaesthesia Sprint Audit of Practice (ASAP) was commissioned by the Healthcare Quality Improvement Partnership and managed by the Royal College of Physicians in partnership with the Association of Anaesthetists of Great Britain and Ireland (AAGBI).