Using a CriSTAL scoring system to identify pre-morbid conditions associated with a poor outcome after admission to intensive care in people 70 years or older

Abstract

Older people admitted to intensive care are considered to have lower physiological reserves, an increased susceptibility to infection and longer recovery times, resulting in generally poorer outcomes after intensive care treatment. However, biological heterogeneity makes identification of those with the best chances of survival within their group difficult and risks subjecting those at the end of their lives to unsuccessful treatments. There is no fit-for-purpose outcome prediction tool capable of identifying patients most at risk of these poor outcomes at the point of admission to intensive care. This retrospective study sought to identify factors associated with mortality in older patients (≥70 years) admitted to a teaching hospital critical care unit using objective variables readily available at the point of admission. A total of 15 variables were tested for a significant association with mortality. Of these, eight were identified as significant variables (myocardial infarction within 6 months, an abnormal ECG, congestive cardiac failure (NYHA ≥2), chronic pulmonary disease, chronic liver disease, metastatic cancer, a stay in hospital ≥5 days preceding ICU admission, and frailty (Clinical Frailty Score ≥4)). These variables were used from the basis of a novel outcome prediction model. The aim of such a model would be that it could be used at the point of referral to intensive care to inform considerations regarding admission, and to facilitate conversations with the patient and family regarding realistic treatment expectations.

Introduction

The UK population aged over 65 years has increased by 21%, and the population aged over 85 years has increased by 31% since 2005.¹ The combination of an older, frailer population who are undergoing major surgery and availability of novel treatment options is resulting in an increased demand for all healthcare services.²

Older patients have less physiological reserve, an increased susceptibility to infection, and consequently much longer recovery times, meaning admission to intensive care (ICU) results in generally poorer outcomes.^3–6 Of patients who survive beyond hospital discharge, many encounter physical disability, depressive symptoms, cognitive impairment, and/or post-traumatic stress disorder in addition to the burden of new or worsened physical illness.^7,8 As a result, ICU survivors, irrespective of age, report lower quality-of-life (QOL) outcomes for all domains at baseline and at 6 months to 14 years after discharge.⁹

Admission to intensive care (IC) can be a life-changing event for any individual: loss of financial stability, ongoing poor health, or inability to live independently are all significant outcomes for those who may have apparently achieved a ‘good’ recovery and outcome. This is even more significant for those who may be nearing the end of their lives and in whom admission to IC and ‘survival’ may be accompanied by a significant change in circumstances after discharge and for the last few months of their life. Previously, ICU outcome prediction models have focused on different numerical age brackets to calculate likelihood of poor response to interventions, however, recently the degree of frailty in an individual has shown to be a more accurate predictor of short- and long-term mortality, the decline in functional independence and a lower quality of life after a stay in ICU.^10–12

Understanding the complex relationship between pre-morbid factors and eventual outcomes in the IC population carries huge significance for giving patients and families useful, relevant information on the burdens and potential benefits of treatment, particularly as they get older. Currently, outcome prediction tools focus only on population-level outcomes by using a combination of pre-morbid factors and acute physiology values to produce a standardised numerical value, which represents the predicted outcome from that set of circumstances.^13–15 However, to date, there is no fit-for-purpose stratification tool for use at the point of admission to IC, which might be used to inform such conversations.

Chronic disease variables, markers of chronic ill health and degree of frailty, have been previously identified as significant predictors of short-term (30-day) mortality for elderly ward patients by the Criteria for Screening and Triaging to Appropriate aLternative care (CrISTAL) study. CriSTAL criteria were identified from a large literature review with the aim to develop a screening tool to identify elderly patients in hospital wards at the end of life and quantify the risk of death in hospital or soon after discharge.¹⁶ The variables in the study were selected due to their simplicity, objectivity, and independence from each patient’s immediate resuscitation status.

The aim of this study was to review the CriSTAL criteria in an IC population to determine whether they are also able to identify patients who are less likely to survive despite ICU admission. It is intended that the final model would be able to offer a starting point to begin discussions with the patient and their relatives about likelihood of survival from an ICU admission and the possibility of alternative treatment options, e.g. end-of-life care.

Methods

The project was registered (CEU project number 7710) as an extension of a previous service review registered and conducted at Sheffield Teaching Hospitals (CEU project number 6825) in September 2016.

A predetermined sample size of 1000 elderly patients (70 years and older) admitted as an emergency to critical care units at the Northern General Hospital (NGH) or the Royal Hallamshire Hospital (RHH) in Sheffield between March 2015 and December 2017 were included in the review. Variables used were adopted from the original CriSTAL scoring system with the aim of creating a new, unique IC scoring system for our unit and included data that are currently routinely collected or noted as part of the patient’s admission and stay in ICU. Fifteen variables from the original model were identified for further investigation in an ICU population. These consisted of ten chronic disease variables (chronic kidney disease (CKD) ≥ Stage 3, myocardial infarction (MI), abnormal ECG prior to admission (any ECG not showing sinus rhythm and/or brady/tachycardia), congestive cardiac failure (CCF), chronic obstructive pulmonary disease, chronic liver disease, cancer, dementia, psychiatric history) and four markers of health (any amount of documented weight loss, stay in hospital ≥5 days preceding ICU admission, ICU readmission during the same hospital stay) and frailty. The Clinical Frailty Score (CFS) was used to grade the varying degrees of frailty. The CFS is an internationally-validated nine-point assessment tool designed to quantify frailty in the critically ill.^10–12 The CFS ranges from 1 (very fit) to 9 (terminally ill). Groups used in this review were CFS grade: ≤3 (No significant frailty), 4 (Vulnerable), 5 (Mildly frail), 6 (Moderately frail), 7 (Severely frail), and 8 (Very severely frail). Each patient admitted to critical care in Sheffield is routinely assigned a CFS based on their overall health status prior to hospital admission and during their hospital stay, and scores are documented in the patient’s record. These scores were used for analysis in this study. Of these 15 independent variables, six were sub-divided into severity groups to attempt to identify the point when certain chronic diseases become significantly associated with mortality in an ICU population. These were CKD (Stages 3, 4, and 5), MI (time of occurrence within 6 months and ≥6 months prior to admission to ICU), CCF (NYHA grades 1, 2, 3, and 4), cerebrovascular disease (transient ischaemic attack and stroke), cancer (isolated solid tumour, local metastasis, and metastatic disease), and frailty (Rockwood CFS grades 4, 5, 6, and 7). Each parameter was assigned a numerical value. In Cr1 all variables were scored 1 point whilst in Cr2 parameters were weighted with more significant variables assigned higher scores (see Online Appendix 1).

All patient level of care was also documented as either level 2 (patients receiving basic respiratory support, e.g. mask/hood CPAP or single-organ support) or level 3 (patients receiving advanced respiratory support, e.g. invasive mechanical ventilation or requiring a minimum of two organs supported). This enabled the two levels of care to be analysed separately, reducing the influence of selection bias.

A retrospective epidemiological database analysis of admission and outcome data was collected from the electronic patient data management system (Metavision). Only ICU-based outcomes were studied whereby survival was indicated by discharge from the unit, whilst mortality was death in the unit. The sample was divided randomly into two equally sized groups (Training and Test) for two different retrospective analyses using SPSS version 23.0 for Windows (IBM Corp. 2014. Armonk, NY: IBM Corp). The purpose of this was to examine the prognostic ability of two modified CriSTAL models (Cr1 and Cr2). The first half of the dataset (Training) would identify significant variables and test the Cr1 model which contained all 15 variables mentioned previously, whereby the absence of a predictor scored 0 points whilst the presence of a predictor scored 1 point, totaling a maximum score of 15. Binary data were analysed with a logistic regression, which was used to determine the mortality odds ratios, along with the degree of significance of each CriSTAL variable on outcome. This analysis was then used to create the new modified CriSTAL model (Cr2), which would then be applied and tested on the second half of the sample (Test). Using a two-sample method meant no patients would be tested twice in an attempt to reduce bias in the model and best replicate its clinical use in different patient cohorts. Due to the small sample used in this study, a p value less than 0.1 was used to determine variables included in Cr2 to formulate a model that included clinically meaningful as well as statistically significant variables causatively included in the Cr2 model. Performing a “Backwards LR” in binary logistic regression function to the training sample, insignificant criteria were eliminated in a stepwise manner and the associated impact on the model’s predictive power calculated. Using mortality odds ratios as a guide, the remaining criteria were weighted appropriately and formed the new modified CriSTAL model (Cr2) (Online Table 1). Final analysis focused on comparing the predictive power of each model and its clinical appropriateness using receiver operator characteristic (ROC) associated AUC values.

Results

Data were obtained from the period 1 March 2015 to 31 December 2016 for a total of 1000 consecutive elderly (70 years and older) patients admitted to critical care, with a mean age of 77.5 years ( ± 5.83 SD; range: 70–101 years). The distribution in levels of care was approximately 3:1 for level 2 and level 3 organ supports, respectively (755 level 2 patients and 245 level 3 patients). In all, 728 (72.8%) patients were alive at discharge and the remaining 272 (27.2%) died in the unit. The total sample was split randomly into two comparable size groups termed Training (mean age: 77.3 ± 5.83 SD; total mortality: 29.2%) and Test (mean age: 77.7 ± 5.83 SD; total mortality: 25.2%) for different analyses. In total, frailty (CFS ≥ 4) was observed in 699 (69.9%) (median = CFS4 (IQR ≤ 3–4), range 1–7) patients, whilst the remaining 301 (30.1%) were deemed to have no significant degree of frailty.

Analysis of the Training sample with the Cr1 model identified eight variables to be clinically important in predicting mortality (p ≤ 0.1) including: MI within 6 months of admission (p = 0.009; OR 2.67; 95%CI 1.28–5.58), abnormal ECG (p = 0.001; OR 2.28; 95%CI 1.42–3.67), congestive cardiac failure (NYHA ≥ 2) (NYHA = 2; p = 0.76; OR 2.43; 95%CI 0.91–6.50), chronic obstructive pulmonary disease (p = 0.026; OR 1.83; 95%CI 1.08–3.12), chronic liver disease (p < 0.001; OR 12.73; 95%CI 3.14–51.62), metastatic cancer (p = 0.21; OR 3.46; 95%CI 1.21–9.93), stay in hospital ≥ 5 days preceding critical care admission (p < 0.001; OR 2.83; 95%CI 1.62–4.97), and frailty (CFS ≥ 4) (CFS 4; p = 0.054; OR 1.86; 95%CI 0.99–3.48). Logistic regression results showed that both models were able to predict population-based outcome to a reasonable degree (Cr1 – 76.6% vs Cr2 – 79.6%). Of these results, specificity was high, whilst sensitivity was comparatively low (Cr1 – 98.3% and 38.4%, respectively, vs Cr2 – 96.2% and 28.2%, respectively).

Both models were able to differentiate patients depending on outcome according to the independent T tests. The survival group tested using Cr1 (mean 2.79; SD 1.52; N = 354) had statistically lower CriSTAL scores than patients who died in the unit (mean 3.65; SD 1.50 N = 156) p < 0.001 (mean difference −0.96; 95%CI −1.25 to −0.67). Similarly using the Cr2 model, survivors (Cr2 – mean 3.72; SD 2.53; N = 374) scored significantly lower than deceased patients who died in the unit (Cr2 – mean 6.20; SD 3.32; N = 126) p < 0.001 (mean difference −2.48; 95%CI −3.29 to −1.84).

Figure 1(a) and (b) represents the two models’ predictive power in the form of ROC curves. Associated AUC results for Cr1 (AUC 0.67; 95%CI 0.62–0.72; p < 0.001) and Cr2 (AUC 0.72; 95%CI 0.66-0.77; p < 0.001) indicate that whilst both are predictive of mortality, Cr1 would be classified as a poor predictive model (AUC 0.6–0.7) whereas Cr2 would be classified a fair prediction model (AUC = 0.7–0.8). Further analysis of the Cr2 model revealed a score of ≥ 4 would correctly identify 73% of dying patients; however, the associated sensitivity trade off suggests that 36.1% of dying patients would be incorrectly classified.

Figure 1.

Receiver operator curves for the Cr1 (a) and Cr2 (b) models. The Cr1 dataset was based on the original CrISTAL dataset. The Cr2 model consisted of fewer variables than the Cr1 dataset.

Discussion

Initial analysis of 500 elderly patients identified eight variables significantly predictive of mortality including moderate-to-severe degrees of frailty, indicating that incorporating frailty into an ICU outcome model was appropriate – a result in agreement with much of the similar literature.^4,12,17–19 Six of these variables already have a large evidence base indicating these would be expected to be significant predictors of mortality in our population. However, two parameters (the presence of any ECG abnormality and a hospital stay ≥5 days preceding admission) were novel, statistically significant indicators in predicting mortality in patients over 70 years of age in our ICU patient population.

Both CriSTAL models (Cr1 and Cr2) expressed statistically significant p values in the independent T tests indicating that CriSTAL scores for patients who survived and died were proven to have absolute differences: both models were therefore able to statistically differentiate patients by outcome. However, the significant mean difference in the Cr1 cohort was only equivalent to 0.96 of a point, a value which clinically cannot be used to effectively separate both groups of patients. The weighting of variables did improve the mean difference between mortality and survival in the Cr2 model, increasing the difference on the groups to 2.48. Whilst both models are statistically appropriate, neither can be suggested at present as clinically useful in any predictive sense simply due to the minimal separation in score for each outcome. However, the benefit of being able to identify poor predictive factors in an ICU population of older patients is helpful when providing information to patients and relatives regarding their views on the likely benefits and burdens of treatment. This was the original aim of our review and adds weight to the concept of conducting conversations on likelihood of survival and therefore potential benefit from ICU admission based on outcome data for the critical care unit.

Logistic regression outputs (Online Appendix 2) were able to calculate the ability to predict population outcome based on the variables in the scoring system. Both models seemed to lack the capability to correctly predict patients likely to die (percentage correct – Cr1 38.4%; Cr2 30.2%) with the later Cr2 model performing notably worse. This indicates the elimination of eight variables from the Cr1 model, whilst classed as insignificant, did contribute to better mortality prediction. Conversely, both models’ ability to predict survival was extremely high (percentage correct Cr1 – 93.8%; Cr2 – 96.3), with Cr2 possessing the higher correct percentage. This analysis found that both models could be clinically useful in helping discussions of likely survival and although eliminating statistically insignificant variables improves the model’s usability and reduces completion time, it does sacrifice accuracy of mortality prediction. However, the high percentage of correct results for survival indicate that patients with no or very few variables of morbidity are extremely likely to survive suggesting that both models might serve better as a survival predictor.

Overall, the calculated predictive capacities of Cr1 and Cr 2 were deemed to be poor (AUC = 0.67) and fair (AUC = 0.72), respectively. Well-validated current ICU mortality predictors such as APACHE IV (AUC = 0.86–0.89), ICNARC (AUC = 0.87), and SAPS III (AUC = 0.80) all possess notably better predictive capabilities than Cr2.^20,21 However, there is evidence that previous versions of APACHE and SAPS underperform when applied to an elderly cohort.²² When compared against models designed and tested for use in the elderly, the predictive power of Cr2 was similar to other larger, more complex, and disease-specific models (AUC = 0.66–0.75).^23–25 The study hypothesis that our final model (Cr2) would accurately predict ICU mortality in a retrospective cohort was not proven to be true, results from the ROC curve indicate that scoring above 4 points would correctly identify almost three-quarters of patients likely to die and when compared to the MPM₀III model (AUC = 0.721-0.80),^20,21 Cr2 exhibited competitive AUC results whilst only containing half the number of variables. Another factor in its favour is usability, as unlike the MPM₀III, the information required is available for use at the point of consideration of referral assuming a medical record is available. This implies Cr2 does have a reasonable level of survival prediction, which could usefully inform conversations with referring teams, patients, and relatives. Furthermore, of the 15 variables identified as significant predictors of elderly ward patient mortality, eight variables, including the finding of two novel parameters (ECG abnormality and hospital stay ≥ 5 days preceding admission), were found to be significant predictors of mortality once the individual was admitted to ICU. Despite identifying numerous predictors of mortality, we failed to achieve strong prognostic sensitivity and specificity when incorporated into a model, indicating some patients would be liable to false-positive and false-negative results. Falsely identifying a patient as likely to survive will not affect any treatment decisions, however, it can provide additional burden and harm to the patient and relatives. Alternatively, the implications associated with incorrectly identifying a patient likely to die can have huge significance on treatment decisions and decisions regarding end-of-life care.

In conclusion, our findings indicate that the Cristal criteria alone could not be used as a clinical decision aid. However, the ability to be able to use them to explore patient and family expectations of survival during conversations regarding admission to intensive care, can be a helpful way of framing those conversations without direct reference to age.

This study demonstrates that using only a few simple chronic disease markers at the point of admission has the potential to predict short-term ICU mortality. Chronic morbidities and frailty are better considerations when making clinical decisions for critically ill elderly patients rather than chronological age alone. These variables should therefore play integral part in any discussions with the patient or relatives regarding realistic treatment expectations and preferences for end-of-life care.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.