Individualized Prediction of Critical Illness in Older Adults: Validation of an Elders Risk Assessment Model

Abstract

Background:

The electronic health record (EHR) presents new opportunities for the timely identification of patients at high risk of critical illness and the implementation of preventive strategies. This study aims to externally validate an EHR-based Elders Risk Assessment (ERA) score to identify older patients at high risk of future critical illness during a primary care visit.

Methods:

This historical cohort study included patients aged ≥65 years who had primary care visits at Mayo Clinic Rochester, MN, between July 2019 and December 2021. The ERA score at the time of the primary care visit was used to predict critical illness, defined as death or ICU admission within one year of the visit.

Results:

12,885 patients were included in the analysis. The median age at the time of the primary care visit was 75 years, with 44.6% being male. 93.7% of participants were White, and 64.2% were married. The median (25^th, 75^th percentile) ERA score was 4 (0, 9). 11.3% of study participants were admitted to the ICU or died within one year of the visit. The ERA score predicted critical illness within one year of a primary care visit with an area under the receiver operating characteristic curve of 0.84 (95% CI 0.83-0.85), which indicates good discrimination. An ERA score of 9 was identified as optimal for implementing and testing potential preventive strategies, with the odds ratio of having the primary outcome in patients with ERA score ≥ 9 being 11.33 (95%CI 9.98-12.87).

Conclusions:

This simple EHR-based risk assessment model can predict critical illness within one year of primary care visits in older patients. The findings of this study can serve as a basis for testing and implementation of preventive strategies to promote the well-being of older adults at risk of critical illness and its consequences.

Introduction

The United States (U.S.) population aged ≥65 years increased by 38.6% over 10 years and reached 55.8 million people (16.8% of the total U.S. population) in 2020.¹ This demographic shift brings challenges due to increased patient complexity, including multiple health issues, functional decline, and mortality risks.^2,3 As the population ages, the demand for critical care services, healthcare utilization, and expenses are expected to rise significantly.^4–6

Annual admissions to intensive care units (ICUs) in the U.S. range from 6 to 8 million patients, contributing substantially to healthcare utilization.^5–7 However, in a cluster-randomized clinical trial, the practice of systematic triage in the emergency department (ED) to ICU has not shown improvement in mortality and quality of life for critically ill older patients compared to routine practice.⁶ Furthermore, ICU survivors often experience a loss of independence and psychological and cognitive issues.^8,9

Community-based interventions hold potential for preventing ICU admissions in the U.S.¹⁰ The COVID-19 pandemic has underscored the need for pragmatic interventions to address actionable risk factors for critical illness early, ideally in the primary care environment. This may help reduce the burden of critical illness on the healthcare system and improve patient outcomes.^11,12

The widespread adoption of the electronic health record (EHR) presents new opportunities for the timely identification of older patients at risk of critical illness by synthesizing EHR-documented risk factors. Numerous prognostic tools are available for the risk stratification in older adults in the community. These tools, including Adjusted Clinical Group, Charlson Comorbidity Index, and Hierarchical Condition Categories, are widely accepted and validated to use for prediction of future hospitalizations, ED visits, and healthcare costs.¹³

The Elders Risk Assessment (ERA) score is a prognostic tool developed at Mayo Clinic and validated for prognosis of hospitalizations, ED visits, critical illness and 2-year mortality in older patients.^3,14–17 In contrast to most of the widely accepted tools, which require data collection not typically performed during routine clinical visits, the ERA score is readily available and visible in the EHR. It can serve as an indicator of older patients at risk for critical illness and mortality at the time of the primary care visit and as the trigger for preventive strategies. The ERA score ranges from −1 to 34 and incorporates age, marital status, the number of hospital inpatient days in the previous 2 years, and comorbidities including diabetes, heart disease, stroke, chronic obstructive pulmonary disease, cancer, and dementia (Supplemental Table 1).¹⁶ While the ERA score is utilized in our primary care practice for implementing and measuring transition programs and facilitating discussions on goals of care, it has not been routinely employed to assess the risks of critical illness and mortality and to serve as a trigger for preventive action during primary care visits.

The objective of this study was to externally validate the ERA score to identify older adults at high risk of critical illness within one year of a primary care visit.

Materials and Methods

This historical cohort study included patients empaneled in the Division of Primary Care Internal Medicine and the Department of Family Medicine at Mayo Clinic Rochester. The Mayo Clinic Institutional Review Board approved the study as exempt (22-009055). The requirement for written informed consent was waived. The TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis): Prediction Model Validation guidelines were used in the conduct of this study as well as in the reporting of our results (Supplemental Table 2).¹⁸

Study Population

The study included patients aged ≥ 65 years who had an office, comprehensive, or telemedicine visit at a primary care or family medicine clinic at Mayo Clinic Rochester, Minnesota, between July 1, 2019, and December 31, 2021, and who had authorized the use of their EHR for research in accordance with the Minnesota state statute.¹⁹

Data Collection

We abstracted demographic data from the time of the primary care visit, including age, sex, race, ethnicity, marital status, comorbidities, ERA score, and the number of hospital days within two years before an index primary care visit. For patients who had multiple primary care visits during the study period, we used the most recent visit as the index event.

Outcome data within one year of the primary care visit were abstracted, including hospital admissions or ED visits, inpatient length of stay (LOS), ICU admissions and LOS, use and duration of invasive mechanical ventilation (IMV), use and duration of non-invasive ventilation (NIV), defined as use of CPAP or BiPAP, and 1-year mortality.

Data retrieval from the EHR was performed using our institutional databases and query tools including ICU DataMart^20,21 and Mayo Data Explorer.²² No imputation methods were required. Additional details about data collection are described in the Supplementary Methods S1.

Predictor Variable

The primary predictor was the ERA score at the time of the patient’s most recent primary care visit, calculated using a previously developed algorithm.¹⁶ Additional details are provided in the Supplementary Methods S1.

Outcomes

The primary outcome was critical illness, defined as a combined outcome of death and/or ICU admission within one year of the primary care visit. Secondary outcomes included 1-year mortality, ICU admission, and combined outcomes including death and/or ICU admission requiring any ventilation (IMV or NIV), and death and/or ICU admission requiring IMV within one year of the primary care visit.

Statistical Analysis

Patient demographics were summarized using median (interquartile range [IQR]) for continuous variables and frequency (percentage) for categorical variables. Precision, recall, and false positive rates were calculated for every score threshold. Discrimination of the score was assessed using the area under the receiver operating characteristic curve (AUROC). Since there are no corresponding probabilities associated with the ERA score for the endpoints in this analysis, score calibration was assessed using 10-fold cross-validation. For each fold, univariable logistic models fit to the out-of-fold data estimated the coefficient for ERA score and were used to estimate event probabilities in the in-fold data. Mean estimated probability and observed event rate with exact 95% binomial confidence intervals (CIs) were summarized according to decile of estimated probability (i.e., 0-9%, 10-19%, etc.) and presented graphically.

Additionally, we determined an optimal ERA score threshold for implementation of potential preventive strategies during the interdisciplinary group discussion. This determination was based on previous literature describing the score thresholds, the burden and feasibility within clinical practice, the specific panel of patients to which it would be applied, the characteristics of the planned intervention,²³ and model performance. Subsequently, we assessed odds ratios (OR) with 95% CIs for experiencing the primary outcome in patients above and below this threshold. P-values <0.05 were considered significant. Data management and analysis were conducted using R version 4.2.2 (R Foundation for Statistical Computing, Vienna, Austria), JMP Pro software (SAS Institute, Cary, NC) and MedCalc for Windows, version 19.4 (MedCalc Software, Ostend, Belgium).

Results

Study Participants

Between July 1, 2019, and December 31, 2021, a total of 13,926 patients aged ≥65 years were seen by their primary provider at Mayo Clinic Rochester, Minnesota. Of these, 12,885 (92.5%) authorized use of their EHR for research¹⁹ and had an ERA score available, making them eligible for inclusion in the analysis. A study flow diagram is presented in Figure 1.

Figure 1.

Study Flow Diagram.

The median patient age at the time of their primary care visit was 75 (69, 83) years, with 44.6% being male. Most participants (93.7%) were White, 64.2% were married, and the median Charlson comorbidity index was 6 (4, 8). Supplemental Figure 1 demonstrates the distribution of the ERA scores in the study population. The median ERA score at the time of the primary care visit was 4 (0, 9). The demographic characteristics of the study participants included in the analysis are detailed in Table 1. The patient characteristics of the included patients did not differ significantly from all eligible patients (Supplemental Table 3). The ERA score was available for 12,885/12,909 (99.8%) of eligible patients, and outcomes were available for all patients.

Table 1.

Baseline Demographics and Clinical Outcomes of Patients Included in the Analysis (N = 12,885)¹

Characteristic / Outcome	N (%) or Median (IQR)
Baseline Characteristics
Age at time of primary care visit, years	74.8 (69.3, 82.5)
65-69	3,663 (28.4)
70-79	5,041 (39.1)
80-89	3,173 (22.5)
≥90	1,008 (7.8)
Sex
Female	7,133 (55.4)
Male	5.752 (44.6)
Race
White	12,072 (93.7)
Black	220 (1.7)
Asian	348 (2.7)
Other	150 (1.2)
Unknown	95 (0.7)
Ethnicity
Not Hispanic or Latino	12,449 (96.7)
Hispanic or Latino	188 (1.5)
Unknown	248 (1.9)
Marital status
Married	8,273 (64.2)
Widowed	2,611 (20.3)
Divorced	1,160 (9.0)
Single	767 (6.0)
In partnership	25 (0.2)
Separated	34 (0.3)
Unknown	15 (0.1)
ERA score at time of primary care visit	4 (0, 9)
Hospital Admission or ED visit 2	6855 (53.2)
Inpatient length of stay days 3	1.26 (0.26, 5.21)
Comorbidities
Diabetes mellitus	2806 (21.8)
Cancer	1990 (15.4)
Cardiovascular disease4	3954 (30.7)
Cerebrovascular accident or stroke	748 (5.8)
Chronic obstructive pulmonary disease	958 (7.4)
Dementia	1371 (10.6)
Charlson Comorbidity Index	6 (4, 8)
Obesity5	6,302 (49.0)
Outcomes 6
Hospital Admission or emergency department visit	4196 (32.6)
Intensive care unit (ICU) admission	493 (3.8)
1-year mortality	1223 (9.5)
Mechanical ventilation	261 (2.0)
Non-invasive	172 (1.3)
Invasive	138 (1.1)
Inpatient length of stay, days	0.97 (0.22, 5.03)
ICU length of stay, days	2.07 (1.03, 4.17)
Days of mechanical ventilation	0.75 (0.23, 2.21)
Non-invasive	0.59 (0.22, 1.27)
Invasive	0.77 (0.22, 2.80)

¹Patients with ERA score available at time of primary care visit

²Within 2 years prior to primary care visit

³Including emergency department, within 2 years prior to primary care visit

⁴Including coronary artery disease, myocardial infarction, and congestive heart failure

⁵36 missing

⁶Within 1 year of a primary care visit

Outcomes

Primary Outcome

The primary outcome, defined as critical illness (death and/or ICU admission within 1 year of a primary care visit) occurred in 11.3% of patients (3.8% had an ICU admission, and 9.5% died). The ERA score demonstrated good discrimination in prediction of critical illness, with an AUROC of 0.84 (95% CI 0.83-0.85).^24–26 In almost all cases, the exact 95% binomial confidence intervals overlapped the identity line in the first 6 deciles of estimated probability, indicating reasonable calibration in that range. However, observed probability tended to be less than predicted in the highest 4 deciles. The precision-recall curves demonstrated that an ERA score of 8 had a positive predictive value (PPV) of 28 and a sensitivity of 0.79. An ERA score of 9 demonstrated a PPV of 31 and a sensitivity of 0.75 for the primary outcome (Figure 2). The ERA score sensitivity and specificity are presented in Supplemental Table 4.

Figure 2.

Receiver-operator characteristic curve, precision-recall curve, and calibration plot for the Elders Risk Assessment score in predicting critical illness, defined as death or intensive care unit admission within one year of a primary care visit. The ERA score demonstrated good discrimination with an area under the receiver operator characteristic curve of 0.84, 95% CI 0.83, 0.85.

Patients with an ERA score ≥ 9 had significantly higher odds of developing critical illness in our study (OR 11.33, 95% CI 9.98-12.87) which is in line with previous research.¹⁵ As described in the methods, following the interdisciplinary team discussion of relevant factors, we found clinically meaningful to choose the ERA score of 9 as a threshold for highlighting the risk of critical illness and as a trigger for initiating of preventive strategies.

Secondary outcomes

Secondary outcomes are presented in Table 1. The ERA score predicted the development of the combined outcome of death and/or any ventilation (IMV or NIV) with the AUROC 0.85 (95%CI 0.85-0.86) (Supplemental Figure 2). Additionally, the ERA score predicted ICU admission only with the AUROC 0.76 (95% CI 0.74-0.78); mortality only with the AUROC 0.86 (95% CI 0.85-0.87). Patients with a higher ERA score had a higher probability of requiring any ventilation within 1 year of their most recent primary care visit (AUROC 0.75 (95%CI 0.73-0.78)), and of receiving IMV (AUROC 0.69 (95% CI 0.65-0.73)). Furthermore, the ERA score predicted the development of the combined outcome of death and/or IMV with the AUROC 0.85 (95%CI 0.85-0.86) (Supplemental Figure 3).

Discussion

In this historical cohort study, the EHR-based ERA score predicted critical illness (defined as death and/or ICU admission) within one year of a primary care visit, with an AUROC of 0.84, indicating good discrimination.^24–26 Additionally, the ERA score predicted ICU admission, 1-year mortality, and the combined outcome of death and/or requirement for ventilation (any or IMV) use. While the model performed well in the majority of probability ranges, there is a room for improvement for better accuracy in predicting outcomes in the high-risk cases. We selected an ERA score of 9 as a threshold for a pilot implementation study of the ERA score with a checklist intervention.²³

Our findings align with previous studies utilizing the ERA score in various contexts and outcomes. These studies reported the ability of the ERA score to predict ED visits and hospitalizations in older adults;¹⁶ mortality and nursing home placement;³ and critical illness defined as death, MV, or sepsis.¹⁵ Espinosa and colleagues validated the ERA score on a cohort of patients seen in the ED and found that patients with higher ERA scores had a higher probability of return ED visits within 30 days, hospital admission within one year of ED visit, and mortality at 30, 90, 180 days, and one year.¹⁷

Notably, the studies evaluating the ERA score with primary care patients^3,14–16 utilized the same cohort of patients assigned to a primary care provider on January 1, 2005, and analyzed the outcomes over a 2-year period ending on December 31, 2006. In contrast, our study externally validated the predictive value of the ERA score for critical illness within a more recent cohort of older primary care patients. In addition, we sought to evaluate more immediate outcomes over a shorter time frame. Our explicit intention was to select an appropriate threshold for subsequent intervention,²³ and the 1-year follow-up period was better suited for further assessing the potential impact of prevention strategies.

Several other prognostic tools exist for risk stratification in older community-dwelling individuals.²⁷ However, there are challenges with these tools including a reliance on self-reported information, limiting their applicability to large populations.²⁷ One commonly used tool for predicting hospital admission and healthcare costs among older individuals is the Probability of Repeated Admission (PRA) score.^28–30 Although PRA demonstrated good discrimination in the meta-analysis conducted by Wallace et al., the low sensitivity makes it unreliable for excluding hospital admission in patients stratified as low risk.²⁹

The demand for critical care continues to grow as the population ages and become more medically complex.³¹ Some ICU admissions may be preventable with early community-based interventions.¹⁰ Therefore, investing in primary care infrastructure to support the prevention of ICU admissions should be considered a valuable strategy alongside the expansion of ICU capacity.¹⁰ In this context, the ability of a simple clinical score to predict critical illness in primary care may be extremely helpful for identifying patients who can benefit most from preventive strategies, potentially reducing the need for ICU admissions.

The adoption of a proactive approach in primary care, along with the implementation of a predictive clinical score, could play a pivotal role in preventing critical illnesses, benefiting patients and optimizing healthcare resource allocation.

Strengths and Limitations

Strengths of our study include its conduct in an academic center with robust informatics infrastructure, a multidisciplinary study team, and the use of a large population-based dataset. Study patients represent both urban and surrounding rural areas. This study serves as the foundation for subsequent research phases, including a pilot implementation study.

Limitations include the retrospective nature of the study using EHR and administrative data, potentially underestimating risk factors. Some outcomes may be underreported if patients received care outside of Mayo Clinic.

The study was conducted within one healthcare system, which may introduce referral bias and limit generalizability to other settings. Access to the ERA score or its parameters may also differ in other healthcare systems, affecting generalizability. For example, marital status may not be recorded uniformly in EHR data in diverse healthcare systems. Furthermore, the race/ethnicity distribution of participants in this study may differ significantly from other regions in the U.S. (93.7% of our sample were white and 96.6% were of non-Hispanic ethnicity). Potentially, other health systems with more diverse populations and other confounding social determinants of health to consider may find this score less predictive or need to reestablish an appropriate cut-off score based on their own population characteristics. External validation may be required prior to clinical use at other institutions or if some of the ERA score components are missing. Also, we evaluated the association of the ERA score from a single event with pre-defined outcomes. Future research may explore the trajectory of the ERA score across a series of events and over time. Additionally, the timeframe of the study overlapped with the COVID pandemic which may have impacted the outcomes.³²

Conclusion

The simple EHR-based risk assessment model can predict critical illness within one year of primary care visit in patients aged 65 years and older. These findings support implementation of the ERA score along with preventive strategies to enhance the quality of outpatient care delivery for older adults at risk of critical illness.

Key points:

• The extensive utilization of the electronic health record provides new opportunities for timely identification of patients at high risk of critical illness and for the development and implementation of potential preventive strategies.

• The simple EHR-based risk assessment model can predict critical illness within one year of primary care visits in patients aged 65 years and older.

Why does this paper matter?

The findings of this study support implementation of the Elders Risk Assessment score along with preventive strategies to enhance the quality of outpatient care delivery for community-dwelling older adults at risk of critical illness.