Prospective Validation of a Checklist to Predict Short‐term Death in Older Patients After Emergency Department Admission in Australia and Ireland

Abstract

Background

Emergency departments (EDs) are pressured environment where patients with supportive and palliative care needs may not be identified. We aimed to test the predictive ability of the CriSTAL (Criteria for Screening and Triaging to Appropriate aLternative care) checklist to flag patients at risk of death within 3 months who may benefit from timely end‐of‐life discussions.

Methods

Prospective cohorts of >65‐year‐old patients admitted for at least one night via EDs in five Australian hospitals and one Irish hospital. Purpose‐trained nurses and medical students screened for frailty using two instruments concurrently and completed the other risk factors on the CriSTAL tool at admission. Postdischarge telephone follow‐up was used to determine survival status. Logistic regression and bootstrapping techniques were used to test the predictive accuracy of CriSTAL for death within 90 days of admission as primary outcome. Predictability of in‐hospital death was the secondary outcome.

Results

A total of 1,182 patients, with median age 76 to 80 years (IRE‐AUS), were included. The deceased had significantly higher mean CriSTAL with Australian mean of 8.1 (95% confidence interval [CI] = 7.7–8.6) versus 5.7 (95% CI = 5.1–6.2) and Irish mean of 7.7 (95% CI = 6.9–8.5) versus 5.7 (95% CI = 5.1–6.2). The model with Fried frailty score was optimal for the derivation (Australian) cohort but prediction with the Clinical Frailty Scale (CFS) was also good (areas under the receiver‐operating characteristic [AUROC] = 0.825 and 0.81, respectively). Values for the validation (Irish) cohort were AUROC = 0.70 with Fried and 0.77 using CFS. A minimum of five of 29 variables were sufficient for accurate prediction, and a cut point of 7+ or 6+ depending on the cohort was strongly indicative of risk of death. The most significant independent predictor of short‐term death in both cohorts was frailty, carrying a twofold risk of death. CriSTAL's accuracy for in‐hospital death prediction was also good (AUROC = 0.795 and 0.81 in Australia and Ireland, respectively), with high specificity and negative predictive values.

Conclusions

The modified CriSTAL tool (with CFS instead of Fried's frailty instrument) had good discriminant power to improve certainty of short‐term mortality prediction in both health systems. The predictive ability of models is anticipated to help clinicians gain confidence in initiating earlier end‐of‐life discussions. The practicalities of embedding screening for risk of death in routine practice warrant further investigation.

The increasing demand for hospital emergency services with the ageing of the world population poses practical, financial, and ethical challenges for the health system.1 The busy and crowded emergency department (ED) is not the most appropriate environment for comprehensive assessment of basic or complex needs of older people with multiple comorbidities, and more geriatric‐friendly services are needed.2 Lack of time or skill to identify patients on the dying trajectory can result in administration of nonbeneficial medical interventions rather than supportive end‐of‐life–oriented management for terminal older patients.3 Older people present with multiple chronic illnesses, polypharmacy, physical vulnerability, and social circumstances that require attention to prevent adverse outcomes and reattendance to health services.

An urgent and more relevant approach to indicate risk of death across several chronic and complex conditions is needed so emergency physicians can identify patients who may require deescalation of treatment rather than potentially harmful treatments or who may present with ambulatory care–sensitive conditions. Risk stratification and prediction is emerging as one such strategy to assist clinicians in discussing decisions such as observation or palliative care referral. However, some prediction tools are gaining widespread use with limited validation,4 and others are used for decision making despite their modest predictive accuracy.5 Baseline population risks alone are insufficient to give clinicians confidence in an individual prognosis as other demographic, disease‐specific, and personal factors may change the course of illness.6

This study estimated a personalized, adjusted risk assessment for older patients presenting at EDs with a view to enhancing prognostic certainty and improving clinician involvement in timely end‐of‐life discussions. The Criteria for Screening and Triaging to Appropriate aLternative care (CriSTAL),7 developed following an extensive literature review on evidence of predictors, is the subject of this validation.

Primary Objective

To establish the predictive ability of individual and combined parameters in the CriSTAL tool to predict short‐term post‐discharge death

Secondary Objectives

The secondary objectives were to determine the minimum number of variables that are sufficient to adequately predict death and to assess the predictive ability of CriSTAL for in‐hospital death

Methods

This prospective cohort study recruited patients aged ≥ 65 years from five Australian teaching hospitals (derivation cohort) and one Irish hospital (validation cohort). Detailed design, recruitment, follow‐up, and analyses are presented in the protocol article, published elsewhere.8

Older patients (aged 65 years or above) presenting at EDs for any nonelective reason who were in the ED for at least one night or admitted to hospital for several days and were able to consent or had a surrogate to do so were eligible to participate. Consent included permission to access their hospital data at baseline and at hospital discharge, conduct a brief interview on admission, and follow up with a phone call to the household approximately 3 months postdischarge. Patients returning home on the same day of presentation were not eligible, due to lack of time to obtain consent and conduct all study procedures. Those with severe cognitive impairment, the critically ill, or those unable to communicate in English were all declared ineligible to participate unless they had a surrogate to enable participation and data collection. Recruitment was conducted until minimum sample of 300 was reached in each participating hospital (July 2015 to March 2016 in Australia and January to March 2016 in Ireland). All patients were assessed by purpose‐trained nurses or junior doctors with the CriSTAL tool, a risk prediction checklist that contains a frailty scale (Fried)9 of a number of chronic conditions, acute parameters present on admission, and history of health service use. Most of the parameters were extracted from the clinical record, and only those missing from the record were obtained via patient or surrogate's interview. Data Supplement S1(available as supporting information in the online version of this paper, which is available at http://onlinelibrary.wiley.com/doi/10.1111/acem.13664/full) shows the variables and point assignment for the score calculation. Follow‐up was up to July 2016 in Australia and to June 2016 in Ireland. Discharge disposition and telephone follow‐up 3 months after discharge were used to ascertain short‐term death. We concurrently measured frailty with the Clinical Frailty Scale (CFS)10 to assess its usefulness and test the feasibility and accuracy in the ED. This was done because of the inherent difficulties of measuring Fried frailty in critically ill people presenting to that setting.

The primary outcome was death within 3 months and CriSTAL's predictive ability. Predictive ability for in‐hospital death was the secondary outcome.

Data Analysis

The Australian cohort was used to derive the prediction rule and then the Irish cohort was used to externally validate the prediction. The derivation cohort was chosen as the tool was developed in Australia. The Irish cohort was used as validation as they expressed interest in the use of the tool in their health system because they saw potential usefulness after seeing the original publication. Accuracy was estimated from regression models conducted for individual variables and combinations of variables based on backward elimination where the outcome was death at 3 months or in‐hospital death. Delays in locating patients for follow‐up meant that some outcomes were ascertained beyond 3 months in Australia. Ten thousand random bootstrap resamples were used to internally validate the models and estimate 95% confidence intervals (CIs). Areas under the receiver‐operating characteristic (AUROC) curve were calculated to determine model discrimination. Calibration was measured by regressing predicted probabilities from the model with the observed values. During the internal validation in Australia using logistic regression directly with CriSTAL score as a summary measure yielded an AUROC of lower accuracy than the model using all the explanatory variables that make up the tool. In the external validation on Irish data, rather than using the summary score we modeled only the association of the CriSTAL components with the outcome, which enhances the utility for clinicians. An acceptable model was defined as AUROC > 70%. Univariable models of all CriSTAL items were assessed initially and likelihood ratio tests were used to assess the statistical significance of model predictors. The predictor selection was based both on statistical grounds and clinical plausibility. All tests of significance used p < 0.05 (two‐sided). All analyses were conducted in SAS version 9.4. The final multivariable model's internal validity was tested using 10,000 random bootstrap resamples to obtain CIs that reflect sampling variability. Sensitivity, specificity, positive predictive value, and negative predictive value of CriSTAL for all‐cause short‐term death(within 3 months of admission assessed the predictive performance. Detailed descriptions of recruitment, follow‐up, analyses, and other processes are presented in the protocol.8

This study was conducted according to the Declaration of Helsinki guidelines, including written consent by patients or surrogates and ability to withdraw at any time. This multicenter study received endorsement from the South Eastern Sydney Local Health District Ethics Committee (#15/026 HREC/15/POW/55) for all the Australian sites. The Irish hospital obtained individual institutional approval from the clinical research ethics committee, University College Cork.

Results

The derivation cohort consisted of 1,143 older people and the validation cohort was 349 older people, with low prevalence of do‐not‐attempt‐resuscitation orders (4.8 and 6.0%, respectively) and advance health directives (0.44 and 0.29% respectively). Patients without a surrogate present could not be recruited due to inability to consent, particularly people with dementia in Australia (Figure 1). Once recruited, the retention rate was high in both cohorts with 130 (11.4%) lost to follow‐up in Australia and 40 (11.5%) in Ireland (Figure 1). Confirmed participants’ mortality at the end of the follow‐up period was 10.1% (116) in Australia and 12.9% (45) in Ireland. The median follow‐up times were 124 days (interquartile range [IQR] = 105–170 days) in Australia and 93 days (IQR = 92–98 days) in Ireland. There were no statistically significant differences in the distribution of age group, sex, baseline frailty levels, number of chronic diseases, or CriSTAL scores between the fully followed up participants and those lost to follow‐up in either cohort.

Figure 1.

Recruitment and follow‐up outcomes.

Forty‐seven percent of Australian participants and 60% of Irish participants had at least one of the target chronic illnesses. The distribution of individual CriSTAL risk factors (Table 1 using the criteria from Data Supplement S1) indicates that the derivation cohort was older, with similar frailty levels (as measured by Fried) but had used hospitals and ICU more frequently in the past year. The validation cohort reported more mental impairment and advanced malignancy and had fewer nursing home residents and longer length of stay but with lower utilization of community support after discharge.

Table 1.

Comparative Risk Factor Profiles of the Derivation (Australia) and Validation (Ireland) Cohorts^*

CriSTAL Variables, n (% of cohort)	Frequency
	Australia, n = 1,143	Ireland, n = 349
Age (years), median (IQR)	80 (73–86)	76 (70–83)
Female	594 (52.0)	185 (53.0)
Length of stay (days), median (IQR)	3.0 (95% CI = 1.0–7.0)	7.0 (3–14)
Nursing home resident	74 (10.3)	20 (5.7)
Advanced malignancy	64 (5.6)	35 (10.0)
Any mental impairment†	123 (10.8)	62 (17.8)
Dementia only	70 (6.1)	35 (10.0)
Proteinuria‡	3 (0.26)‡	1 (0.29)‡
Chronic kidney disease	133 (11.6)	47 (13.5)
Fried frailty score > 3	357 (31.2)	115 (33.0)
Congestive heart failure	146 (12.8)	41 (11.8)
Chronic obstructive pulmonary disease	179 (15.6)	59 (16.9)
New or previous myocardial infarction	126 (26.7)	41 (11.8)
New cerebrovascular accident	16 (1.4)	10 (2.9)
Chronic liver disease	19 (1.7)	4 (1.2)
Hypoglycemia	9 (0.8)	0 (0.0)
Low urinary output	16 (1.4)	15 (4.3)
Abnormal ECG	457 (40.0)	140 (40.0)
Abnormal oxygen saturation	193 (16.9)	75 (21.5)
Meet > 2 RRS criteria§	70 (6.1)	28 (8.0)
Hospital admission in the past year	671 (58.7)	184 (52.7)
ICU admission in the past 12 months	88 (7.7)	10 (2.9)
Two or more chronic conditions	175 (15.31)	62 (19.8)
Community services postdischarge	220 (19.3)	25 (7.2)

Data are reported as number (% within cohort) unless otherwise specified.

^*Australia—deceased by end of study n = 116, survivors n = 1,027; Ireland—deceased by end of study n = 45, survivors n = 304.

†Mental impairment includes one or more of the following: dementia, long‐term mental illness, disability from stoke, or acute behavioral changes.^‡Information on proteinuria 59% missing in Australia and 70.8% missing in Ireland.

^§RRS criteria to call an emergency team to rescue inpatients who are deteriorating on general floors. Includes abnormal changes in blood pressure, pulse, respiratory rate, oxygen saturation, blood sugar levels, urinary output, temperature, or level of consciousness.

ECG = electrocardiogram; ICU = intensive care unit; IQR = interquartile range; RRS = Rapid Response System.

Other risk factors were similar in the two cohorts (Table 1). Females in both cohorts were significantly more frail than males (35.5% vs. 26.9%, χ² = 9.2, p = 0.024 in Australia; and 38.6% vs. 27.0%, χ² = 4.9, p = 0.027 in Ireland). Females were also generally older than males (mean = 81 vs. 79 years, p < 0.001 in Australia; and mean = 78 vs. 75 years, p < 0.01 in Ireland).

The mean aggregated CriSTAL scores was significantly higher for the deceased than for the survivors (p < 0.0001; Figure 2), with mean scores, respectively, of 8.1 (95% CI = 7.7–8.6_ versus 5.7 (95% CI = 5.1–6.2) in Australia and 7.7 (95% CI = 6.9–8.5) versus 5.7 (95% CI = 5.5–6.0) in Ireland. A CriSTAL score of >7 appeared to be a suitable cut‐point for high risk of short‐term death in the derivation cohort as the association at this point was highly statistically significant (77% of deceased and 38% of survivors, χ² = 65.85 and p < 0.0001). In the validation cohort, the trend appeared to be less clear cut and the threshold lower (Figure 2). The association was most significant at scores of >6 (80% of the deceased and 20% of the survivors, χ² = 13.6 and p < 0.001).

Figure 2.

Distribution of CriSTAL scores in the two populations.

Unadjusted analysis of CriSTAL risk factors suggested significant association (p ≤ 0.01 for odds ratios [ORs]) of short‐term death and age, sex, nursing home residence, frailty, dementia, abnormal respiratory rate or oxygen saturation, acute behavioral disorders, and hospitalization in the past 12 months in the derivation cohort. In the validation cohort, significant unadjusted associations with short‐term death were the same, with the exception of previous hospitalization and the addition of chronic liver disease and abnormal electrocardiogram.

Table 2 shows the results for the final multivariable models, after backward elimination. For the derivation cohort, older age, male sex, and advanced malignancy were significant predictors of short‐term death whether frailty was measured with Fried or CFS. By contrast, in the validation cohort male sex was no longer a significant predictor and age was only a significant predictor in the model when frailty was measured with Fried. Oxygen saturation was a significant predictor in both validation models, while frailty was only significantly associated with short‐term death when measured with CFS.

Table 2.

Final Multivariable Modeling of Short‐term Death Using Two Frailty Instruments Within the CriSTAL Tool for Derivation Cohort (Australian) and Validation Data Alone (Irish) and Based on Logistic Regression, Bootstrap Resampling

	Model With Fried Frailty Scale				Model With Rockwood Frailty Scale
aOR	95% Wald Confidence Limits		p‐value	aOR	95% Wald Confidence Limits		p‐value
Effect in derivation cohort
Intercept	0.007	0.003	0.014	<0.001	0.002	0.0005	0.004	<0.0001
Age	1.04	1.01	1.07	0.012	1.04	0.99	1.07	0.0385
Male	2.19	1.39	3.62	0.001	2.45	1.50	4.20	0.0008
Advanced malignancy	5.91	2.89	12.25	<0.001	4.92	2.19	10.82	<0.001
Nursing home residence	3.12	1.61	5.91	0.001
Frailty as Fried	2.15	1.75	2.75	<0.001
Frailty as Rockwood					1.97	1.46	1.81	<0.001
AUROC	0.825 (0.784–0.869)				0.81 (0.76–0.86)
Effect in validation cohort
Intercept	0.075	0.029	0.15	<0.0001	0.002	0.0001	0.02	<0.0001
Age	1.04	1.00	1.10	0.0290	1.03	0.98	1.08	0.1699
Male	1.08	0.52	2.24	0.8264	1.02	0.49	2.05	0.9553
Oxygen saturationa	2.95	1.45	6.11	0.0023	2.45	1.20	5.10	0.0111
Frailty as Fried	1.23	0.95	1.65	0.0999
Frailty as Rockwood					1.80	1.30	2.92	0.0002
AUROC	0.700 (0.626–0.801)				0.77 (0.71–0.85)

^aAbnormally low and meeting calling criteria for rapid response call (SaO₂ < 90%).

aOR = adjusted OR from multivariable analysis.

The AUROC (Figure 3) showed good discriminant ability for short‐term mortality in all models using either Fried or CFS frailty scales in both settings but the accuracy was higher for the derivation cohort. The model discrimination was higher for the validation cohort when Frailty was measured using CFS.

Figure 3.

Prediction of 3‐month mortality: comparison of AUROC for derivation and validation cohorts using two frailty instruments. AUROC = area under the receiver‐operating characteristic.

The final model performed better in the Irish cohort when CFS (Rockwood) was incorporated. At lower probabilities of death (≤25%), the final model with CFS is more sensitive for Ireland (42.2% vs. 32.1%) but more specific for Australia (89.1% vs. 95.8%; details in Data Supplement S1, Table S2). At probabilities of 50% and above the model is highly specific for both cohorts even though CFS frailty was the only common predictor in both cohorts.

The discriminant ability of CriSTAL incorporating CFS instead of Fried's frailty score for the secondary outcome of in‐hospital mortality prediction was also good for Ireland and Australia (AUROC = 0.810 and 0.799, respectively; details in Table S2), with similar predictor profile to the 3‐month models. The derivation and validation models for in‐hospital death prediction had optimal sensitivity at predicted probabilities of 5% and best specificity at predicted probabilities of >10%. The mean length of stay was longer for the deceased in hospital than for the hospital survivors but not significantly different in the validation cohort, likely due to small numbers (1.04 days vs. 5.8 days, respectively, in Australia, p < 0.0001; and 18.2 days vs. 11.4 days in Ireland, p = 0.058).

Discussion

This validation of a risk prediction tool based on personalized objective clinical criteria available at the point of care showed that frailty (as assessed with CFS) remained the single most significant and stable predictor of short‐term death among older patients. In addition, older age, male sex, and malignancy contributed to the risk in Australia, whereas impaired oxygen saturation on admission increased the odds in Ireland. The model with higher discriminant ability in both settings (AUROC = 0.77 in Ireland and 0.81 in Australia) incorporated the CFS frailty instrument. The models were highly specific at predicted probabilities of ≥25%, which is useful when clinicians want to rule in chances of death and decide at which point to initiate end‐of‐life discussions (Data Supplement S1, Table S1). The positive predictive value for short‐term death was higher in the validation cohort possibly due to higher death rates. Conversely, the negative predictive value was consistently high in both cohorts, reassuring of the reliability of the true negatives (Table S1).

In the derivation cohort, after adjusting for age and frailty males, had twice the odds of death as females. This effect was not present in the validation cohort, but after confounders were adjusted for, low oxygen saturation more than doubled the odds of death. By comparison with the derivation cohort who was recruited over 6 months and two seasons, recruitment in Ireland occurred throughout the three winter months and this may have impacted results in the validation cohort. As per the Irish findings, oxygen saturation < 90% on admission to ED and advanced age also predicted in‐hospital death in a model applied to a large sample of older people presenting to EDs in Switzerland.11

CriSTAL's discriminant ability for short‐term death prediction for older ED patients has been reasonably consistent in three countries (Australia,12 United States,13 and Denmark14) in terms of predictor variables, with the exception of the variable nursing home residency. In Ireland, we attempted to validate predictors to explore applicability in a vastly different health system. CriSTAL performance was also good with only a few risk factors being significant predictors of death after multivariable adjustments. Fried's frailty was not significantly associated with short‐term death, and this may reflect a cultural difference in self‐report. The younger age of the Irish cohort and lower rate of nursing home usage might indicate that the study population in Ireland was different from that in Australia and Denmark; i.e., the possibility exists that that Irish cohort included those who would be managed in community services in other countries and therefore are on an earlier point in the trajectory toward death than those studied in Denmark or Australia.

The CriSTAL score differentials for deceased and survivors where an uneven distribution are apparent in the validation cohort compared with the smoother trend in the derivation cohort, possibly due to the smaller sample size (Fig. 2). The validation cohort had lower representation of very old patients (18% aged 85+ years vs. 31.9% in the derivation cohort). Length of stay for them was significantly longer than in Australian sites, a reflection of the lack of availability of community services for patients to be discharged to in Ireland. The validation cohort also had fewer nursing home residents and reported lower use of community support, thus reflecting differences in the operation of the two health systems, admission criteria, and/or differences in hospital case mix. The Irish health system has substantially fewer hospital beds15 explaining delayed discharge from acute hospitals due to lack of community places, less resourced primary care,16 and community support services, which could also explain lower number of nursing home residents.

A modified version of CriSTAL tested in the United Kingdom also found that metastatic cancer and Rockwood's frailty were significant predictors of death among older patients in intensive care.17 Our findings are also consistent with those of another prospective study in Dutch EDs, where cognitive impairment was associated with physical decline and 90‐day mortality (AUROC = 0.79, 95% CI = 0.73–0.85).18 Risk stratification of shorter or longer mortality intervals for a broad range of diseases is becoming feasible, particularly using retrospective data sets, but its prospective applicability at the point of care was yet to be demonstrated. A simple risk score using 16 parameters has been used in Ireland before with high discrimination for earlier time frame to death (30‐day mortality, AUROC = 0.85); older age and a frailty proxy were significant predictors (OR = 1.04 and 2.8, respectively) in common with our study.19

On the more relevant prediction of 3‐month mortality, CriSTAL in Australia and Ireland performed better than other recent reports from in the Netherlands, Japan, and United States. A large prospective Dutch study of emergency patients ≥ 70 years found predictive discrimination for mortality was similar to ours in the derivation cohort (AUROC = 0.79, 95% CI = 0.73–0.85), but lower in the validation cohort (AUROC = 0.67, 95% CI = 0.60–0.73).20 As in our study, predictors of 90‐day mortality were increasing age (OR = 1.59), nursing home residence (OR = 2.08), and patient needing help with dressing (OR = 3.60), while being a female had a protective effect (OR = 0.25).

The Japanese study also yielded lower performance than CriSTAL at AUROC of 0.774 for death within 3 months, with sensitivity of 69% and specificity of 74%. The identification of Seniors at Risk tool has been reported to have a sensitivity of 85% and specificity of 37% for 3‐month mortality.5 In our study CriSTAL had the highest sensitivity (>84.0%) and specificity (56.2%) for short‐term mortality in the derivation cohort (Data Supplement S1, Table S1). The corresponding values for the validation cohort were 84.0 and 54.6%, respectively.

The usefulness of an objective risk prediction tool for older people lies in the opportunity to yield a global assessment that encompasses their multimorbidity, frailty, and history of recent health service use, to help plan the most appropriate care pathway including palliative care referrals. In our study, the longer length of stay of those dying in the hospital suggests either that patients suffered complications that prevented appropriate discharge or that there is a need for improved management where patients could be transitioned elsewhere earlier for end‐of‐life care. This summary score may also help in assessing other composite outcomes to guide postdischarge care planning to respond to increasing health system pressures.21 Yet, it is acknowledged that there are mixed views on the use of numeric prognostic scores to support decision making. Some clinicians are reluctant to use them due to lack of understanding of the technical properties or appropriate time to use, while others welcome the opportunity to cross‐validate their intuitive judgment.22

If screening for clinical prediction for patients near the end of life in the ED by ED staff was feasible, this could inform the development of clinical guidelines for transition to nonactive care pathways where safe. Screening for frailty and risk of death in over 1,000 people by a purpose‐trained nonspecialist nurses or junior doctors was feasible in five EDs as it took on average 5 minutes to document parameters. However, these staff were not existing ED personnel but called upon for a limited time to conduct this task. Also, screening and recruitment only occurred during business hours due to resource constraints. Whether the practice can sustainably be embedded in routine practice is yet to be tested.

Risk prediction is complex and tends to perform better for specific patient types, e.g., surgical or selected conditions such as cancer. The CriSTAL tool was designed to assist in the identification of risk for older frail patients with complex multiple comorbidities to flag people who would benefit from timely end‐of‐life discussions about potential care pathways. The accuracy level adds utility to the clinician at the point of care, but as previously proposed by others, risk prediction is only an adjuvant to clinical decisions before patient discharge.5 However, predicting death for people with various concurrent conditions of different trajectories6 remains a challenge. A subsequent study examining risk stratification for these older ED patients to distinguish high from medium‐ or low‐risk patient subgroups could complement this validation for more practical guidance. Keeping in mind the characteristics of the health system in which it is used, a model of service in EDs could incorporate screening for high risk of death to prompt conversations on appropriate care pathways possibly preventing hospitalization. This is consistent with the Choosing Wisely campaign to enhance understanding of goals of care and prevent potentially harmful further testing or treatment.23 In Ireland, the validation could be improved with a future larger study of consecutive patients across multiple hospitals to attain a representative sample.

Limitations

This validation study across two diverse patient populations and geographic regions addresses “shortcoming of existing risk‐stratification methods.”24 Strengths were the large sample size with high retention rates, similar distribution or risk factors between the full completers and those lost to follow‐up, and the harmonization of the protocol. Among the limitations, our estimation of frailty was self‐reported and covering the week prior to admission via the ED rather than frailty at the time of the acute episode leading to the presentation. The exclusion of patients mentally incompetent to consent or those critically ill who did not have a surrogate available reduced our ability to examine the predictability of CriSTAL on those subpopulations. Given the known association between those two risk factors and death, our results can only be considered an underestimate. The delays in follow‐up for most of the Australian cohort meant that short‐term prediction was on average at 4 months rather than at 3 months as initially intended. There are advantages and disadvantages in attempting comparisons extrapolating from one health system to another. We found that administering the CriSTAL parameters was feasible in the ED, but applying the Australian algorithm to Irish data did not yield an optimal fit. The choice of validation cohort was convenience following the Irish collaborators request. Finally, it is clear that factors beyond patients’ clinical profile not measured in this study may have influenced the outcome and should be examined further.

Conclusion

The CriSTAL risk prediction tool is feasibly administered in EDs and performs reasonably well with high specificity and negative predictive value in predicting short‐term and more modestly in predicting in‐hospital death of older people presenting for a nonelective reason. A minimum of five of 29 variables were sufficient for accurate prediction, and a cut point of 7+ or 6+ depending on the cohort was strongly indicative of risk of short‐term death. Frailty was the single most stable predictor of short‐term death across models and intervals. The use of Clinical Frailty Scale instead of the Fried score as part of CriSTAL is recommended in future, as Rockwood's Clinical Frailty Scale score has performed better across contexts. Increased accuracy should reassure clinicians on the identification of patients with high risk of death presenting in emergency to trigger discussions on preferences and goals of care. We recommend incorporating hospital case mix indicators or health system practices to test whether this improves the relevance and predictive ability of the models. The practicalities of administration of risk screening in routine practice by core ED staff warrant further investigation.

Supporting information

Data Supplement S1. Supplemental material.

Academic Emergency Medicine 2019;26:610–620.