Optimal screening for increased risk for adverse outcomes in hospitalised older adults

Abstract

Background: screening for frailty might help to prevent adverse outcomes in hospitalised older adults.

Objective: to identify the most predictive and efficient screening tool for frailty.

Design and setting: two consecutive observational prospective cohorts in four hospitals in the Netherlands.

Subjects: patients aged ≥70 years, electively or acutely hospitalised for ≥2 days.

Methods: screening instruments included in the Dutch Safety Management Programme [VeiligheidsManagementSysteem (VMS)] on four geriatric domains (ADL, falls, undernutrition and delirium) were used and the Identification of Seniors At Risk, the 6-item Cognitive Impairment Test and the Mini-Mental State Examination were assessed. Three months later, adverse outcomes including functional decline, high-healthcare demand or death were determined. Correlation and regression tree analyses were performed and predictive capacities were assessed.

Results: follow-up data were available of 883 patients. All screening instruments were similarly predictive for adverse outcome (predictive power 0.58–0.66), but the percentage of positively screened patients (13–72%), sensitivity (24–89%) and specificity (35–91%) highly differed. The strongest predictive model for frailty was scoring positive on ≥3 VMS domains if aged 70–80 years; or being aged ≥80 years and scoring positive on ≥1 VMS domains. This tool classified 34% of the patients as frail with a sensitivity of 68% and a specificity of 74%. Comparable results were found in the validation cohort.

Conclusions: the VMS-tool plus age (VMS⁺) offers an efficient instrument to identify frail hospitalised older adults at risk for adverse outcome. In clinical practice, it is important to weigh costs and benefits of screening given the rather low-predictive power of screening instruments.

Introduction

Hospitalisation is a hazardous event for older adults. The incidence rate of subsequent adverse outcomes such as functional decline and loss of independence has been reported to be high [1–5] and often permanent [6, 7]. Many studies have been conducted to find a powerful screening tool to identify older patients at an increased risk for adverse outcomes. The predictive performances of these tools have been summarised and compared in reviews concluding that none of the instruments investigated had a strong predictive power and could be considered a ‘gold standard’ [8–12]. In addition to predictive performance, efficiency, feasibility in clinical practice and face-validity were considered important for a screening instrument [12, 13].

As part of the transitional intervention project ‘Recovery Care Programme’ (HerstelZorgProgramma), we aimed to implement a systematic approach to identify frail older patients in an early stage of hospitalisation. The current article reports on selecting a screening model for the identification of frail older adults who were at an increased risk for adverse outcomes. In Dutch Hospitals, systematic screening of older patients on undernutrition, ADL limitations, falls and delirium [14] has been nationally implemented as part of a mandatory programme (VeiligheidsManagementSysteem = Safety Management Programme) (VMS) since 2012. The results of the screening on these four important geriatric domains were tested for their feasibility to predict adverse outcomes. Subsequently, we tested whether other instruments previously described in the literature improved the predictive power of the screening algorithm.

Methods

Participants

From 1 October 2010 to 1 February 2011 older adults aged ≥70 years, who were admitted to one of four participating hospitals, were approached to participate in the study. In three hospitals, elective and acute patients at the orthopaedic, neurology, urology and surgical units participated. In one hospital, only acute patients were approached to participate, independent of the unit they were admitted to. Patients were eligible if they stayed in the hospital for at least two consecutive days and if they were interviewed within 72 h after admission. Patients with severe cognitive impairment [Mini-Mental State Examination (MMSE) <19] were excluded if no informal caregiver was available for a hetero-anamnesis.

From 1 October 2011 until 1 February 2012, a second cohort was included, applying the same methods and criteria, to validate the results. In this cohort, a short interview was administered, including only questions needed for the assessment of the screening tool established using the original data.

Before the start of the study, the medical ethical committee of the Leiden University Medical Center decided no formal ethical assessment of the protocol was necessary because the aim of the study was to evaluate the quality of care for (frail) older people. All the patients received detailed written information before they were interviewed, and could thereafter refuse to participate. Patient data were anonymised immediately after collection.

Measurements

Trained nurses administered an interview on demographic information, the unit of admission and type of admission (acute/elective) and the instruments that are described below. Three and twelve months after their admission date, patients were sent a self-administered questionnaire to measure their level of functioning.

Screening instruments

The six-item Katz Index on Independence in Activities in Daily Living [15] was used to assess functional status. Patients were asked whether they needed help bathing, dressing, toileting, transferring from bed to a chair and eating and whether they used incontinence materials (yes/no). A patient was considered dependent if having a score of ≥2. The risk for falls was assessed with a single question on whether the patient fell in the last six months (yes/no). A patient was considered to be (at increased risk to become) undernourished when having either lost weight unintentionally and/or having experienced a decreased appetite and used supplemental drinks or tube feeding. A patient was considered to be at risk for delirium if one or more of three questions was answered with ‘yes’: memory problems, the need for help with self-care and previously experienced confusion. These four instruments (on ADL, falls, undernutrition and delirium) make up the Dutch VMS screening programme. We composed a VMS total score by adding up the patients' dichotomised screening results on the four domains, resulting in a score from 0 to 4.

The Identification of Seniors at Risk (ISAR) [16] screening instrument, developed to identify older patients at risk for a functional decline in emergency departments, was assessed. Its six items concerning the need for help with self-care and household activities, previous hospital admission in the last 6 months, vision, memory and polypharmacy add up to a sum score of 0–6 and a score of ≥2 implicates an increased risk.

Two cognitive tests were assessed, the 6-item Cognitive Impairment Test (6CIT) and the MMSE. The 6CIT [17] includes one memory, two attention and three orientation questions, resulting in a score ranging from 0 (best) to 28. Patients with a score >10 are considered to have cognitive impairment. The MMSE items cover the cognitive functions orientation, memory, attention and calculation, language and the ability to follow simple comments [18]. The maximum score is 30 and patients with a score of ≤23 are considered to have cognitive impairment.

Outcome

The outcome was the occurrence of an adverse outcome within 3 months after hospital admittance. Adverse outcome was defined by a decline of ADL function and/or a high-healthcare demand or death. Functional decline was considered present if patients reported one or more additional ADL dependencies on the Katz Index after 3 months of follow-up. Healthcare demand was assessed using information obtained from the healthcare insurer on the patients' indicated budget for long-term care. Patients were considered to have a high-healthcare demand if being indicated a budget that gives access to a sheltered living area with intensive care. Deaths were checked in the systems of the hospital and the healthcare insurer or were reported by family members after receiving the follow-up questionnaire. Adverse outcomes were assessed again after 12 months of follow-up.

Analyses

Candidate determinants for inclusion in the prediction model were selected based on their relative risks for adverse outcome after 3 months of follow-up. The sensitivity, specificity and predictive power of these candidate predictors were calculated in association with adverse outcome. Next, classification and regression trees (CART) [19] were composed that divided the sample in subgroups as homogeneous as possible with regard to the outcome by selecting the most predictive variables based on minimum prediction error. First, only determinants that are readily available of every patient admitted in Dutch hospitals were used to compose a classification tree: age, type of admission (acute/elective) and the data included in the VMS system. The other candidate predictors were subsequently added to the model, and finally they were all added at the same time. Based on these models, high- and low-risk groups were composed; patients in branches of the tree with a risk for adverse outcomes that was higher than the background risk in the total study population were considered to be at ‘high risk’. Calculations of the sensitivity, specificity and power to predict adverse outcomes after 3 months of follow-up were used to compare the different models. Analyses were performed using SPSS, version 20.0 (IBM Corp., Armonk, NY, USA).

This work was supported by the Netherlands Organization for Health Research and Development (ZonMw). The funding organisation had no role in the design or conducts of the study, nor in the data collection and analyses, the interpretation of the data or the realisation process of the manuscript.

Results

Supplementary data available in Age and Ageing online, Figure S1 show details on the inclusion and drop-out in both the initial and the validation cohort. Baseline characteristics of both cohorts are displayed in Table 1. In Table 2, the relative risks for adverse outcome and the predictive capacities of the candidate predictors are displayed. Apart from sex, all determinants were statistically significantly associated with an adverse outcome. The percentage of the population that scored positive on the different determinants ranged from 13 to 72%. The sensitivity of the tests varied from 24 to 89%, the specificity from 35 to 91%, and the AUC from 0.58 to 0.66 over the different predictors. Age and two or more positive scores on the VMS domains were the strongest predictors of an adverse outcome.

Table 1.

Characteristics of patients included in the initial and validation cohort for longitudinal analyses

	Initial cohort	Validation cohort
N	883	812
Sex (% male)	46.0	45.8
Age [median (IQR)]	78 (9)	78 (10)
Acute admittance (%)	45.2	41.4
Length of stay in days, [median (IQR)]	6 (6)	5 (6)
Medical specialty (%)
General surgery	36.7	28.1
Orthopaedics	27.3	38.9
Neurology	14.0	14.2
Urology	15.5	14.7
Other	6.2	4.3
VMS undernutrition* (%)	27.7	27.6
VMS delirium (range 0–3, % ≥1)	55.2	44.3
VMS ADL limitations (range 0–6, % ≥2)	23.3	15.7
VMS falls (%)	37.6	36.4
VMS sum score [median (IQR)]	1 (2)	1 (2)
MMSE (range 0–30, % ≤23)	15.9	–
6CIT (range 0–28, % >10)	13.2	–
ISAR (range 0–6, % ≥2)	57.1	–

IQR, inter-quartile range; VMS, VeiligheidsManagementSysteem, Safety Management Programme; ADL, activities of daily living; MMSE, Mini-Mental State examination; 6CIT, 6-item Cognitive Impairment Test; ISAR, Identification of Seniors at Risk.

^*VMS undernutrition (lost weight and/or both less appetite as well as energy supplement use).

Table 2.

Bivariate associations of all determinants with adverse outcome after 3 months of follow-up and their individual power to predict an adverse outcome^a after 3 months of follow-up

Determinant	RR	95% CI	% of population screened positive	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)	AUC
Female sex	1.08	0.87–1.33
Age (≥80 versus <80 years)	2.17	1.77–2.68	38	62	70	44	83	0.66
Admittance type (acute versus elective)	1.89	1.53–2.33	43	58	63	37	80	0.61
VMS undernutrition	1.93	1.57–2.38	27	40	78	40	78	0.59
VMS delirium (≥1)	2.85	2.18–3.73	47	62	59	37	80	0.61
VMS ADL limitations (≥2)	2.19	1.77–2.70	16	28	88	46	77	0.58
VMS falls	1.97	1.59–2.44	38	54	68	38	80	0.60
VMS total ≥1	3.36	2.28–4.95	72	89	35	33	90	0.62
VMS total ≥2	2.77	2.18–3.52	43	67	66	42	84	0.66
MMSE (≤23)	2.19	1.77–2.71	15	30	90	53	78	0.60
6CIT (>10)	2.12	1.70–2.65	13	24	91	50	77	0.58
ISAR (≥2)	2.02	1.57–2.61	57	73	49	34	83	0.61

RR, relative risk; CI, confidence interval; PPV, positive-predictive value; NPV, negative-predictive value; AUC, area under the (receiver-operating characteristics) curve. VMS, VeiligheidsManagementSysteem, Safety Management Programme; ADL, activities of daily living; MMSE, Mini-Mental State Examination; 6CIT, 6-item Cognitive Impairment Test; ISAR, identification of seniors at risk.

^aAdverse outcome is functional decline, death or high-healthcare demand.

The tree displayed in Figure S2, Supplementary data available in Age and Ageing online, was composed by the performance of a CART analysis including age, type of admittance and all information extracted from the VMS screening. Placing the strongest predictor of adverse outcome in the top of the tree, the population was split firstly by age (70–80 versus ≥80 years). For the second split, the number of positively screened domains of VMS was selected in both branches of the tree. When splitting the branch of patients aged 70–80 years, the optimal cut-off of this VMS total score was ≥3. For the patients aged ≥80 years, the cut-off value of ≥1 created two optimally homogenous groups for the risk for an adverse outcome. The predictive capacities of this model are displayed in Table 3.

Table 3.

Power to predict an adverse outcome^a and its components after 3 and 12 months of follow-up using the instruments of the Dutch Safety Management Programme VMS in combination with age^b

	RR	95% CI	% of population screened positive	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)	AUC
Initial cohort
Three-month follow-up	3.53	2.79–4.46	34	68	74	50	86	0.71
Twelve-month follow-up	3.16	2.46–4.05	34	66	72	44	86	0.69
Validation cohort
Three-month follow-up	2.60	2.15–3.15	35	61	75	57	78	0.68

VMS, Veiligheids Management Systeem = Safety Management Programme; PPV, positive-predictive value; NPV, negative-predictive value; AUC, area under the receiver-operating characteristics curve; RR, relative risk; CI, confidence interval.

^aAdverse outcome is functional decline, death or high healthcare demand.

^bPatients were considered frail when aged 70–80 years and positive on three or more of the four VMS domains or aged over 80 years and positive on one or more domains.

When adding other instruments to the model, only the MMSE had enough predictive power to be included in the regression tree: patients aged over 80 years, without a positive score on the VMS domains were considered positive if they scored ≤23 on the MMSE. This combination was only present in three patients in our population, so the extra step in the decision tree did not change the measures of predictive power.

Table 3 also shows that the screening model predicted adverse outcome after 12 months of follow-up with a sensitivity of 66%, a specificity of 72% and a predictive power of 0.69. In the validation cohort, a sensitivity of 61%, a specificity of 75% and a predictive power of 0.68 were found.

Discussion

We studied the capacity to predict an adverse outcome 3 months after hospitalisation in older adults of several screening instruments for geriatric conditions and demographic information separately, as well as in all possible combinations. The predictive power of the individual instruments was similarly poor, the sensitivities, specificities and percentages of patients screened positive widely differed. Combining age and instruments screening for (the risk for) undernutrition, ADL limitations, falls and delirium (domains of VMS) improved the power to predict adverse outcomes. Adding other instruments to the model did not further improve the prediction. The best predictive power for adverse outcome was found identifying patients at risk when aged 70–80 years and scoring positive on three or four VMS domains; or when aged ≥80 years and scoring positive on one or more of the VMS domains. This VMS plus age (VMS⁺) screening tool showed both an acceptable sensitivity (68%) and specificity (74%), without the majority of patients screening positive (34%).

Several studies and reviews previously investigated instruments to predict adverse outcomes in hospitalised older patients. The power to predict adverse outcomes using several commonly studied instruments, such as ISAR, KATZ-ADL and SHERPA, was shown not to differ much [8–12, 20, 21] and no instrument met the criteria of a gold standard. Few studies [9–11] previously studied several screening instruments within the same population, but to our knowledge, combinations of instruments have not been studied before. No previous studies used the (VMS) screening instruments on four main geriatric problems to predict adverse outcome after hospitalisation. The separate domains did not satisfactory predict the incidence of adverse outcome, but the number of positively scored domains added to the predictive capacity.

In an additional subgroup analysis, we separated acutely and electively admitted patients because the results of the screening can have distinctive implications for the management of care in these groups. The VMS⁺ screening appeared to be highly sensitive (82%) in the acute patients and highly specific (80%) in the elective patients. This increased sensitivity and specificity came at the cost of a decreased specificity and sensitivity, respectively. The validity of performing the screening in elective patients before admission in order to be able to plan preventive measures warrants future research. The low percentage of false-positive results of the VMS⁺ screening in electively admitted patients gives hope that the instrument might be applicable for that purpose.

By using the full-original content of the previously validated instruments, we enhanced the face validity of the screening model. The use of instruments that have been shown to be feasible for implementation [22] is a strength of our study. The results can be easily implemented in Dutch hospitals by integrating the model in electronic medical record systems and using the data collected within the context of VMS. Another important strength is the availability of results of 12-month follow-up measurements as well as validation in a new cohort. Because the interview was drastically shortened in the validation cohort, the participation rate was enhanced. The comparison of variables available in both cohorts showed that the comparability of both cohorts was not affected. The fact that the study population was mostly recruited from surgical and neurological units is a limitation of the study. Conclusions on the performance of the suggested instrument in other patient groups, such as patients admitted to general internal medicine units, cannot be drawn. Future research should focus on validating the results in this group of patients and other populations.

The instrument suggested in the current paper had both an acceptable specificity and sensitivity. The advantage of these test characteristics is that the number of patients unnecessarily screened positive remains manageable, while the number of frail older patients that were missed is acceptable. Because of the very similar predictive powers and the widely differing sensitivities and specificities of the (combinations) of existing screening instruments, a screening instrument must not be chosen based on its predictive power only. The gains and costs need to be weighed carefully. Using a sensitive instrument can cause positive-screening results outnumbering the (financial) capacity of follow-up interventions in a clinical setting. The option to use a validated instrument that is more specific, but less sensitive might be a better solution than, for example, adapting the cut-off value of an instrument to decrease the number of positive results. Another option would be to implement a second step in the screening procedure once patients have been screened positive. When using a very specific instrument, extra attention is needed for the early identification of increased vulnerability in patients during hospital stay. When no other instruments are available or implemented, age can be used as a marker for frailty, as we found age to be a strong predictor for adverse outcomes.

Concluding, the VMS-tool plus age (VMS⁺) offers an efficient instrument to identify frail hospitalised older adults at risk for adverse outcome. Combining information on the separate domains of the Dutch patient safety programme led to a fair risk-stratification. Identifying patients at risk for adverse outcome is only a first step in its prevention. Thorough and reasoned consideration with respect to the specificity and the sensitivity of a screening instrument in relation to the capacity available in clinical practice is required to make responsible choices. It is important to accept that the perfect screening instrument detecting older adults at risk does not yet exist.

Key points.

• The incidence rate of adverse outcomes after hospitalisation is high in older patients.

• Screening hospitalised older patients for frailty can help to prevent adverse outcomes.

• A combination of screening instruments on four geriatric domains (ADL, falls, undernutrition and delirium) offers an efficient instrument to identify frail older patients.

• Patients are considered frail if aged 70–80 years and three or four of the domains are positive, or if aged over 80 years and one or more domains are positive.

• Given the limited predictive power, costs and benefits of screening need to be carefully weighed in clinical practice.

Conflicts of interest

None declared.

Supplementary data

Supplementary data mentioned in the text are available to subscribers in Age and Ageing online.