Evaluating frailty scores to predict mortality in older adults using data from population based electronic health records: case control study

Abstract

Background

recognising that a patient is nearing the end of life is essential, to enable professional carers to discuss prognosis and preferences for end of life care.

Objective

investigate whether an electronic frailty index (eFI) generated from routinely collected data, can be used to predict mortality at an individual level.

Design

historical prospective case control study.

Setting

UK primary care electronic health records.

Subjects

13,149 individuals age 75 and over who died between 01/01/2015 and 01/01/2016, 1:1 matched by age and sex to individuals with no record of death in the same time period.

Methods

two subsamples were randomly selected to enable development and validation of the association between eFI 3 months prior to death and mortality. Receiver operator characteristic (ROC) analyses were used to examine diagnostic accuracy of eFI at 3 months prior to death.

Results

an eFI > 0.19 predicted mortality in the development sample at 75% sensitivity and 69% area under received operating curve (AUC). In the validation dataset this cut point gave 76% sensitivity, 53% specificity.

Conclusions

the eFI measured at a single time point has low predictive value for individual risk of death, even 3 months prior to death. Although the eFI is a strong predictor or mortality at a population level, its use for individuals is far less clear

Introduction

Improving the experiences of those nearing the end of life is a global public health imperative [1, 2]. Early identification of this stage of an individual’s life is an essential step to achieving this goal: patients who are identified earlier have the opportunity to discuss preferences and make advance plans for care [3–5]. Frailty has been identified as a common condition associated with death in community-dwelling older people [6] and reflects a state of increased vulnerability to poor resolution of homoeostasis after a stressor event. This state is associated with an increased risk of adverse outcomes, including falls, delirium, disability, care home admission, hospitalisation and mortality [7–10]. In high income countries approximately 11% of people over 65 years and 25–50% of those over 85 years have frailty [11].

Recognising frailty and its extent in clinical practice may be challenging [12]. An electronic frailty index (eFI) has been developed using electronic primary healthcare records in England to help clinicians identify patients who are living with frailty [10]. The index uses a cumulative deficit model [13] to calculate a frailty score based on a range of symptoms, diagnoses and observations recorded by family physicians. From July 2017, GP practices in England will be required to identify and monitor patients with moderate and severe frailty using a validated frailty instrument. As the eFI is now available on the desktops of a majority of general practitioners in England, it is likely to be widely used to identify patients with frailty.

Increasing levels of frailty are strongly associated with risk of mortality when measured at a single point in time [14–16], but there has been little research investigating the utility of measures of frailty to predict mortality in individuals. This is important, because clinicians need to anticipate death in order to target palliative and end of life care resources. In this study, our aim was to test the hypothesis that the eFI generated from routinely collected data, can be used to predict mortality at an individual level.

Method

Setting

This study used electronic health record data from ResearchOne, a health and care research database containing de-identified clinical and administrative data from approximately six million active electronic healthcare records (EHRs). ResearchOne extracts anonymised data from the SystmOne clinical information system, which is used in over 2500 primary care practices in England. General practitioners use SystmOne to record their consultations (including patient histories, clinical observations, diagnoses, treatments and referrals) with free text and the Read code classification system [17].

Study design

In this historical prospective case control study, probability of mortality was determined using eFI scores calculated 3 months prior to recorded month of death in decedents and 3 months prior to 1 January 2016 for matched survivors. This 3-month window was selected to maximise the ability of the eFI to discriminate between decedents and survivors, whilst still allowing clinicians sufficient opportunity to intervene in patients’ end of life management.

Participants

ResearchOne identified records of individuals (decedents) age 75 and over who died between 01/01/2015 and 01/01/2016. This age group was selected because the study aim was prediction of mortality to inform the need for palliative and end of life care, and the majority of deaths occur in this age group. Furthermore, it has been established that older people are less likely to access specialist palliative care services [18]. A comparison group (survivors) was constructed by identifying patients matched to decedents by age, sex and practice location, but with no record of death between 01/01/2015 and 01/01/2016. Data were not extracted on individuals with records available for fewer than 6 months, and where cause of death was classified as an external cause of mortality (International Classification of Diseases codes version 10). Due to the method of sampling, controls were matched at age of death of the case (between 01/01/2015 and 01/01/2016), but their data was collected on 01/01/2016, when they were known to have not have died during the sampling period; this caused the controls to be on average 6 months older than the cases at measurement.

Test methods

The eFI is a cumulative deficit measure of frailty that calculates a frailty score based on 36 deficits, drawn from a pool of 2000 clinical Read codes for symptoms, signs, diseases, disabilities and abnormal laboratory test values [10]. An individual’s eFI score is calculated by dividing the number of deficits present by the total possible to create a score between 0 (no deficits) and 1 (36 deficits). Severity categories (0–0.12 = fit; >0.12–0.24 = mild frailty; >0.24–0.36 = moderate frailty; >0.36 = severe frailty) are defined using quartiles, with the 99^th centile as the upper limit [10].

Analysis

The ResearchOne dataset was split at random without replacement into a development dataset of 70% of cases and a validation dataset of 30% of cases.

Characteristics of the development and validation dataset cohorts were calculated as frequencies and univariate analyses were used to examine group differences. Unconditional logistic regression was used to examine the association between severe frailty (eFI > 0.36) and mortality in the development dataset, adjusted for matching variables age and sex. To enable predictions appropriate for the complete population, survivors were reweighted (using inverse probability weights calculated using Office of National Statistics life tables for 2013–16) to reflect the population size adjusting for deaths.

Weighted and unweighted receiver operating characteristic (ROC) curves were created and exploratory sensitivity analyses were used to determine an optimum cut point for frailty associated with an increased risk of death, with a target sensitivity of 75%. We chose to propose that the specificity of the test was less important as the questions about end of life care would be acceptable even to those who do not die within a short time. The optimum cut point was then tested in the validation dataset. All sensitivity analyses were repeated stratified by sex and age using 74–84, 85–94, 95+ age strata. All analyses were controlled for the matching variables, with age mean centred. An alpha level of <0.05 was used to signify conventional statistical significance. Analyses were performed using R, CRAN version 3.3.2 [19] (PRROC [20], Survey [21])

Results

Participants

In total, 13,149 decedents age > 75 were identified by ResearchOne and matched to 13,149 survivors. Table 1 contrasts demographic characteristics of the 9,204 decedents in the development dataset and the 3,945 decedents in the validation dataset versus matched survivors. The development dataset contained 4,116 (44.7%) males and 5,088 (55.3%) females. The mean age was 85.1 (SD 6.0) years for decedents and 85.7 (SD 6.0) years for survivors, as expected by design. Mean eFI was significantly higher (P < 0.0001) for decedents (0.29, SD 0.11) than for survivors (0.25, SD 0.11), and mean eFI for females overall (0.28, SD 0.12) was significantly higher (P < 0.0001) than for males (0.25, SD 0.11).

Table 1.

Characteristics of individuals in the development and validation datasets. Decedents at 3 months prior to death and their matched survivors at the same time point.

	Development sample		Validation sample
	Decedents	Survivors	Decedents	Survivors
Age: years, mean (SD)	85.1 (6.0)	85.7 (6.0)	85.1 (6.0)	85.6 (6.0)
Gender
Male, n (%)	4,116 (44.7)	4,116 (44.7)	1,723 (43.7)	1,723 (43.7)
Female, n (%)	5,088 (55.3)	5,088 (55.3)	2,222 (56.3)	2,222 (56.3)
Overall eFI: mean (SD)	0.29 (0.11)	0.25 (0.11)	0.29 (0.11)	0.24 (0.11)
Male eFI: mean (SD)	0.28 (0.11)	0.23 (0.11)	0.28 (0.11)	0.23 (0.11)
Female eFI: mean (SD)	0.30 (0.11)	0.26 (0.12)	0.30 (0.11)	0.26 (0.11)
Frailty category
Not frail, n (%)	563 (6.1)	1,245 (13.5)	262 (6.6)	545 (13.8)
Mild, n, (%)	2,517 (27.3)	3,386 (36.8)	1,043 (26.4)	1,478 (37.5)
Moderate, n (%)	3,355 (36.5)	2,856 (31.0)	1,461 (37.0)	1,216 (30.8)
Severe, n (%)	2,769 (30.1)	1,717 (18.7)	1,179 (29.9)	706 (17.9)

Test results

Development sample

An unweighted logistic regression model with ‘not frail’ as the reference category (Table 2), showed that increasing severity of frailty is strongly associated with higher odds of mortality (severe frailty OR 4.30 95%CI 3.84–4.89). The inverse probability weighted logistic regression model (Table 2) showed a similar strong association between the frailty category and odds of mortality (severe frailty OR 4.72 95%CI 4.16−5.36).

Table 2.

The association (odds ratio, OR) between severity of frailty and mortality 3 months prior to death: results from logistic regression models in the development sample.

	OR2	95% CI	P	OR3	95% CI	P
1Mild frailty	1.76	[1.57−1.97]	<0.0001	1.80	[1.60−2.02]	<0.0001
1Moderate frailty	3.00	[2.68−3.37]	<0.0001	3.19	[2.83−3.59]	<0.0001
1Severe frailty	4.30	[3.84−4.89]	<0.0001	4.72	[4.16−5.36]	<0.0001

¹Reference category is ‘not frail’; ²unweighted; ³ inverse probability weighted.

In an unweighted ROC the area under the curve was 0.62; adjusting for population size with the weighted ROC analysis the area under the curve was 0.69 (Supplementary Figure 1 available at Age and Ageing online). Severe frailty as a predictor of mortality had a sensitivity of 23% (95% CI 22–24%) and a specificity of 91% (95% CI 91–91%). In age and sex specific analysis, optimum cut points were proposed in the range of mild severity (0.17–0.22) with higher values for the females and for older age strata (Table 3).

Table 3.

Analysis of eFI as a screening test for mortality in the validation dataset at 3 months prior to death, stratified by age and sex using cut points identified in the development dataset with a target sensitivity exceeding 75%.

Group	Development data	Validation data
	eFI Cutpoint	Sensitivity	Specificity	Apparent prevalence	True prevalence	Positive predictive value	Negative predictive value
All individuals: all ages	>0.19	0.76	0.53	0.49	0.07	0.11	0.97
Females: 75–84 years	>0.17	0.79	0.46	0.55	0.04	0.05	0.98
Females: 85–94 years	>0.22	0.75	0.42	0.60	0.12	0.14	0.93
Females: 95+ years	>0.22	0.76	0.28	0.74	0.29	0.30	0.74
Males: 75–84 years	>0.17	0.74	0.49	0.52	0.05	0.07	0.97
Males: 85–94 years	>0.19	0.80	0.42	0.62	0.15	0.19	0.92
Males: 95+ years	>0.19	0.91	0.14	0.88	0.37	0.39	0.73

Validation sample

In the validation sample, across all strata these cut points overestimate the prevalence of death and identify over 50% of the sample as being at risk of death where the true values for prevalence lie in the range of 4–37% (Table 3).

Discussion

This study has shown that a single frailty measure has a low predictive value for mortality at an individual level, even close to death. Although the eFI is a strong predictor or mortality at a population level, its use for individuals is far less clear and our findings emphasise the need to understand the application of individual measures of frailty, if they are in widespread use in primary care. Using our proposed optimal cutoff to predict mortality in individuals would overestimate the number of individuals at risk of dying, and could lead to inappropriate targeting of resources.

There are a growing number of initiatives to increase awareness of frailty and improve patient outcomes [22, 23]. In England, a requirement to identify adults with moderate and severe frailty is being introduced into the GP contract from the middle of 2017. This will require an easy to use tool to identify patients with frailty nearing the end of life, and predict their likely future care needs. Several studies have demonstrated an association between frailty and risk of mortality and increased service utilisation [10, 24, 25, 26]. However, these studies have not examined the predictive utility of frailty scores on an individual basis. This study builds on previous findings by focussing specifically on frailty in the last year of life in an unselected primary care population, using a measure of frailty that is routinely available.

Strengths and limitations

The use of large primary care cohort is a major strength of this study. Data in ResearchOne are demographically and geographically representative of the population in England. Because the data are recorded by general practitioners as part of routine care, many of the limitations of survey data, such as cost, non-response or attrition due to ill health, are overcome.

The study design as implemented by ResearchOne generated the potential pool of controls at a fixed time point (controls had to be alive on 01/01/2016), rather than identify a control when every case died between the study period. This design created an imbalance of age where the controls were older than the cases, however as frailty increases with age, and age was used as an adjustment this is unlikely to have affected the results. In addition, no individual could be both a control and then later a case due to the independence of the two sampling mechanisms.

Study exclusion criteria were applied at the point of data extraction and no data were available on the numbers of individuals who were not eligible for study entry. We were unable to comment on instances where no eFI score was available, but we expect this to be a negligible number.

Previous studies suggest that cumulative deficit models of frailty have better predictive power than phenotypic models [27–29]. The eFI has been validated in a large population using two different electronic healthcare record systems [10], but we cannot be sure that our findings would be replicated with a different frailty index.

Recent work has examined longitudinal annual changes in frailty and identified distinct trajectories of frailty associated with higher levels of healthcare utilisation [30]. These findings suggest the possibility that longitudinal changes in frailty scores could be used to target individuals at risk of hospitalisation or death. Future studies should investigate whether distinct trajectories of frailty exist over a shorter time frame, and whether these trajectories could help to indicate to physicians where a patient may have palliative or end of life care needs, improving the specificity without jeopardising the sensitivity.

Implications for practice

For individuals, single time point frailty scores alone are not a strong predictor of mortality, even 3 months prior to death. Further work is needed to examine whether longitudinal change in frailty scores can better predict end of life care needs in older adults.

Key points.

• There is a strong association between severity of frailty and mortality.

• Few studies have attempted to determine the predictive value of frailty scores for mortality at an individual level.

• We have shown that a single frailty score, calculated close to death has low predictive value for mortality in older adults.

• Understanding of the application of individual measures of frailty is essential, if they are in widespread use in primary care.

Supplementary data

Supplementary data mentioned in the text are available to subscribers in Age and Ageing online.

Conflicts of interest

A.C. led the development and validation of the electronic frailty index, which is licensed by the University of Leeds to suppliers of electronic health record systems at no cost on the basis that a premium charge is not then applied to the end user. A.C. and S.D.B. have received travel and accommodation expenses for invited presentations on the electronic frailty index. All other authors declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous 3 years, no other relationships or activities that could appear to have influenced the submitted work.

Ethical approval

Approval for this study was granted by the ResearchOne ethical review panel with oversight from the UK NHS Health Research Authority and the UK Government Health and Social Care Information Centre Confidentiality Advisory Group. ResearchOne was approved by the UK National Health Service (NHS) National Research Ethics Service (11/NE/0184).

Funding

This work was supported by the National Institute for Health Research (NIHR) School for Primary Care Research (SPCR, 2016DS to D.S.); the Medical Research Council (MC_U105292687 to F.E.M.); the National Institute for Health Research (NIHR) under its ‘NIHR Research Professorship’ scheme (Reference Number NIHR-RP-011-043 to L.R.); and by the National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care, Yorkshire & Humber (NIHR CLAHRC to B.H. and A.C.). The research was funded by the NIHR School for Primary Care Research (SPCR). The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health. The funders had no role in design and conduct of the study; collection, management, analysis and interpretation of the data; preparation, review and approval of the manuscript; or the decision to submit the manuscript for publication.

Patient involvement

Members of VoiceNorth (a Newcastle University funded organisation that aims to capture the public’s views on research and policy developments) were involved in this research. A number of them had been caregivers for a relative or friend at the end of life. Their views helped to shape the research questions.