Longitudinal validation of an electronic health record delirium prediction model applied at admission in COVID-19 patients

Abstract

Objective

To validate a previously published machine learning model of delirium risk in hospitalized patients with coronavirus disease 2019 (COVID-19).

Method

Using data from six hospitals across two academic medical networks covering care occurring after initial model development, we calculated the predicted risk of delirium using a previously developed risk model applied to diagnostic, medication, laboratory, and other clinical features available in the electronic health record (EHR) at time of hospital admission. We evaluated the accuracy of these predictions against subsequent delirium diagnoses during that admission.

Results

Of the 5102 patients in this cohort, 716 (14%) developed delirium. The model's risk predictions produced a c-index of 0.75 (95% CI, 0.73–0.77) with 27.7% of cases occurring in the top decile of predicted risk scores. Model calibration was diminished compared to the initial COVID-19 wave.

Conclusion

This EHR delirium risk prediction model, developed during the initial surge of COVID-19 patients, produced consistent discrimination over subsequent larger waves; however, with changing cohort composition and delirium occurrence rates, model calibration decreased. These results underscore the importance of calibration, and the challenge of developing risk models for clinical contexts where standard of care and clinical populations may shift.

1. Introduction

People who are hospitalized with COVID-19 develop wide ranging neuropsychiatric symptoms [[1], [2], [3], [4], [5]] including the acute confusional state of delirium [[6], [7], [8], [9]], which has been shown to occur in 10 to 50% of COVID-19 patients [7,8]. More generally, delirium is the most common neuropsychiatric syndrome encountered in the general hospital setting and a condition of particular concern to psychiatrists working with medical and surgical patients [[10], [11], [12]]. Delirium is associated with a wide range of adverse outcomes including increased critical care utilization, longer length of stay, increased rate of institutional discharge, worse functional outcomes, increased rate of readmission, and increased mortality [[13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34]]. In addition to poorer clinical outcomes, care of patients with delirium is associated with greater caregiver and clinician burden [[35], [36], [37], [38], [39], [40], [41], [42]]. Fortunately, delirium can be prevented through multicomponent interventions [[43], [44], [45], [46], [47]]; however, optimal allocation of these multicomponent prevention resources, as with any other scarce resource, requires a way of identifying individuals at greatest risk for experiencing delirium [48].

The COVID-19 pandemic, with its associated increased demands on caregivers and hospital clinical resources, has increased the potential consequences of a case of delirium, as critical care services and individual caregivers are already overburdened. It has also increased the difficulty of delivering multicomponent interventions that could reduce delirium risk, by necessitating additional infection control precautions and protective equipment conservation efforts [[49], [50], [51]]. This combination of circumstances, which developed during care of the various surges of COVID-19 patients, increased the importance of zero-contact stratification of delirium risk.

Although delirium is often underdiagnosed and under coded, electronic health records (EHR) data can be used to study both the epidemiology and biology of delirium [[52], [53], [54], [55], [56], [57], [58], [59]]. Building on this potential, EHR data produced during routine care have been repurposed for secondary use in developing clinical prediction models of delirium risk [[60], [61], [62], [63], [64], [65], [66], [67]]. We previously reported the development and contemporaneous external validation of a machine learning model of delirium risk among patients with COVID-19 presenting during the spring of 2020 [68]. As the characteristics, treatments, and outcomes of COVID-19 have changed over the course of the pandemic, and clinical model predictive accuracy can vary over time, we sought to replicate this model using the later waves of pandemic patients. In other words, we aimed for a replication which is longitudinal with respect to the patients included in the study but not with respect to an individual patient [[69], [70], [71], [72], [73], [74]]. This work builds on a readily adopted and freely available model, applicable during emergency department care, based on facts already present in the EHR prior to inpatient hospitalization. It seeks to evaluate the extent to which the predictive quality of the model remained stable over successive waves of pandemic care despite evolving patient characteristics and treatment approaches.

2. Materials and methods

2.1. Conceptual approach

We have previously shown that EHR data can be used at the time of hospital admission to predict delirium diagnosis over the subsequent hospital stay in patients with COVID-19. This prediction is possible using a statistical model developed through machine learning which makes a prediction based on diagnostic history, lab test results, medication orders, and other patient characteristics. This development occurred during the first wave of COVID-19 cases; however, over the intervening waves of COVID-19 cases, the composition of patients and treatment approaches changed a great deal [74,75]. The aim of this study was to re-validate the model in these subsequent waves of patients to establish whether the quality of statistical predictions had decreased. The overall methodological approach below outlines who was studied, how the data in their health records were made presentable to the statistical model, and finally the multiple approaches taken to establishing the quality of the predictions made by the statistical model.

2.2. Cohort development

This study included all adults hospitalized with PCR-confirmed SARS-CoV-2 between June 1, 2020, and April 30, 2021, across six hospitals, including two academic medical centers and four community hospitals. The original development of the previously published model occurred during the first surge of patients and included those hospitalized prior to May 31, 2020. This validation is longitudinal in that it occurs in a cohort who were cared for after – in calendar time – those in whom the model was originally developed and validated. As such, it provides an estimate of how the model would have performed had it been deployed clinically during this time. While this study is longitudinal with respect to cohort and evolution of the pandemic, it is only minimally longitudinal with respect to the individual patients included as the span of individual time considered is merely that between emergency room presentation and hospital discharge.

All data for this study—including the demographic, laboratory, diagnostic, medication, and other clinical data—were extracted from the health system EHR [68]. These included date of birth for calculation of age at admission, home zip code for area deprivation index calculation, body mass index (BMI), and lifetime smoking status [76]. The study protocol was approved by the Mass General Brigham Human Research Committee. No participant contact was required in this study which relied on secondary use of data produced by routine clinical care, allowing waiver of informed consent.

2.3. Clinical data handling and feature encoding

To make statistical predictions from the data in the EHR, those chart-like clinical facts must be systematically reshaped into spreadsheet-like data which are suitable for predictive modeling. This reshaping of EHR data into spreadsheet-like data used the same steps as those of the predictive model's initial development and validation [68]. The 34 features – columns in the notional spreadsheet – included in the predictive model (which was used in exactly the previously published form without any modification of any kind) cover a range of diagnostic codes, medications, laboratory results, and registration characteristics. Those features and the coefficients, which come from the original model development and are what would be required to implement this model, are included in Supplemental Table 1. The included features are notable for the relative paucity of protective factors (that is features which are associated with a lower risk of delirium) relative to risk factors (features which are associated with greater risk of delirium when present). Clinical features associated with reduced predicted risk of delirium include serum albumin and body mass index. Clinical features associated with increased predicted risk include laboratory reported high troponin and low absolute lymphocyte count. Of the 34 total features included in the model and presented in Supplemental Tables 1, 16 features are based on current lab tests results, 6 are based on prior diagnosis, 8 are based on medication prescription, and the remainder are patient and demographic characteristics (e.g., age or smoking history).

For this replication study, as in the original development and validation study, BMI, lifetime smoking status, and age at admission were all drawn from predefined structured fields within the EHR. Prior clinical diagnostic codes (both billing or non-billing problem list) were aggregated from native International Classification of Disease (ICD) codes to the second level of the Healthcare Utilization Project (HCUP) Clinical Classification Software (CCS) hierarchy as log-transformed counts of the total number of codes assigned within that category [77]. Medications were encoded as at the Unified Medical Language System (UMLS) RxNorm ingredient as the log-transformed count of orders and prescriptions over the 30 days preceding admission [60,78,79]. For clinical laboratory test results, both continuous features in laboratory supplied units and laboratory supplied flags (logical true or false variables) of abnormally high and low values were extracted as required by the model from the first occurrence within the episode of care (i.e., the first emergency room collection of a given test for those who had multiple instances of the test prior to admission). For count encoded features, medications or diagnostic codes which did not occur were counted zero times. Missing continuous values (e.g., laboratory values) were replaced through medium imputation from the original training sample and thus a prediction was possible on every admission over the study period.

To facilitate cohort description and evaluation of model performance within clinical and demographic substrata, we extracted additional clinical and demographic data from the EHR, including data which occurred after prediction time. These data were not made available to the original model training and played no part in prediction; thus, these evaluative features were unable to contaminate the antecedent predictions (sometimes referred to as ‘peeking’ forward in time [80]). Instead, these features were merely used in secondary analysis of predictive accuracy. Given the importance and frequency of delirium in the critical care setting and delirium superimposed on dementia, dementia diagnosis and subsequent critical care were selected [[81], [82], [83], [84], [85]]; however, it is notable that the former could conceptually be excluded at the time of prediction based on prevalent dementia diagnosis whereas the latter eventual incident cases of critical illness could not. To evaluate model performance over demographic strata, we extracted race and gender from structured EHR demographics tables. Recognizing the clinical importance of critical illness and cognitive impairment in delirium, we extracted both subsequent need for critical care and a previously described cognitive impairment feature from the EHR [86].

For our delirium outcome, we extracted coded delirium diagnosis made during the subsequent hospital admission using the same coded definitions used in initial model development and also applied a natural language processing (NLP) delirium definition to the discharge summaries of the hospital admission as a sensitivity analysis on the coded definition [54,57]. The full bag of codes used to define delirium based on the Medicare General Equivalence Mappings was: F01.51, F03.90, F05, F06.0, F06.1, F06.2, F06.30, F06.4, F06.8, F10.231, F15.920, F19.921, F19.939, F19.950, F19.951, F19.97, G92, G93.40, G93.41, G93.49, R40.0, R40.4, and R41.82. The NLP-based definition produced equivalent results to the primary analysis done against the coded delirium definition and thus these duplicative results are not shown. Conceptually, it is important to note that within a given patient the timing of features used can be divided into three distinct eras of chart time: (1) features used for the statistical prediction of subsequent delirium risk (all of which were available at the time of admission; e.g. emergency department lab tests and past diagnostic history), (2) the clinical diagnosis of delirium used as the reference outcome in evaluating predictive accuracy (which was only available at the time of hospital discharge), and (3) information which played no part in the prediction or outcome but was merely used for post hoc secondary description of predictive accuracy in specific subgroups (e.g. need for critical care).

2.4. Model evaluation and statistical analysis

We characterized the COVID-19 patient cohort and contexts in which patients received care using descriptive statistics (i.e., counts and percentages for discrete variables, means and standard deviations for continuous measures) stratified by eventual diagnosis of delirium. Thereafter, we evaluated the discrimination – the extent to which those who went on to develop delirium were consistently predicted as higher risk than those who did not – and calibration – the extent to which all those predicted to have a given risk of delirium did develop delirium at the predicted rate – of delirium risk predictions made at the time of admission by reference to the presence or absence of an eventual coded diagnosis of delirium over the course of the subsequent hospitalization. We evaluated model discrimination, both in the pooled cohort and by clinical and demographic strata, using area under the receiver operating characteristic (ROC) curve [87]. We evaluated model calibration using the quantile-by-quintile comparison of predicted and observed outcome rate, Spiegelhalter's z statistic, and inspection of calibration plots including both logistic and flexible loess curves [[88], [89], [90], [91] ]. We also calculated the familiar measures of diagnostic testing accuracy (e.g. sensitivity, specificity, negative predictive value, and positive predictive value) by dichotomizing risk predictions at the high-risk threshold identified in the initial model validation cohort [92]. To quantify the expected benefits from intervention in any given case we completed a decision curve analysis of this cohort [[93], [94], [95] ]. Parallel sensitivity check analysis using the NLP augmented delirium outcome identified only a small number of additional cases and the prediction results were consistent with the primary analysis done by reference to the coded diagnosis definition and thus the duplicative NLP results are not shown. All analysis was conducted using R version 4 [96].

3. Results

The cohort included 5102 patients of whom 716 were diagnosed with delirium (14%). The mean age on admission was 62.7 years old (SD 18.73) and 2520 (49.4%) patients were less than 65 years of age on admission. The cohort included 285 (5.6%) patients with a prior diagnosis of dementia and 2525 (49.5%) were treated at community hospitals. Full demographic and clinical characteristics of the cohort are shown in Table 1 .

Table 1.

Sociodemographic and clinical characteristics of the patient cohort stratified by delirium diagnosis.

	Overall	Delirium case	Non-case
n	5102	716	4386
Male gender (%)	2603 (51.0)	406 (56.7)	2197 (50.1)
Race (%)
Black	614 (12.0)	78 (10.9)	536 (12.2)
Other	1259 (24.7)	165 (23.0)	1094 (24.9)
White	3229 (63.3)	473 (66.1)	2756 (62.8)
Age (decade, %)
<30	304 (6.0)	15 (2.1)	289 (6.6)
30s	461 (9.0)	18 (2.5)	443 (10.1)
40s	444 (8.7)	21 (2.9)	423 (9.6)
50s	804 (15.8)	66 (9.2)	738 (16.8)
60s	1000 (19.6)	141 (19.7)	859 (19.6)
70s	1036 (20.3)	181 (25.3)	855 (19.5)
80+	1053 (20.6)	274 (38.3)	779 (17.8)
Prior dementia history (%)	285 (5.6)	112 (15.6)	173 (3.9)
ICU care (%)	795 (15.6)	255 (35.6)	540 (12.3)
Community hospitals (%)	2525 (49.5)	293 (40.9)	2232 (50.9)

The area under the ROC curve (AUC) for the COVID-19 delirium model in this longitudinal replication cohort was 0.75 (95% CI 0.73–0.77; Fig. 1, left), similar to that previously reported for this model (AUC 0.75 [95% CI 0.71–0.79] [68]). In this replication cohort, the observed and predicted prevalence of delirium differed significantly in quantile-by-quintile comparison of predicted and observed outcome rate (χ ²(23) = 56.05, p < 0.001). Spiegelhalter's Z was significant (Z = −2.29, p = 0.02) and calibration plot showed lower expected outcome rates in those with high predicted risk (Fig. 1, right). As such, in this model those who did develop delirium likely had higher risk scores whereas those patients at a specific risk score were less likely to go on to develop delirium at exactly that predicted rate. Model predictions were broadly consistent across clinical and demographic contrasts (Fig. 2, Supplemental Table 2); however, predictive discrimination was generally worse in high-risk patients 65 years and older (AUC 0.67 95% CI 0.64–0.70) and those with a history of dementia (AUC 0.58 95% CI 0.52–0.65).

Fig. 1.

Overall receiver operating characteristic (ROC) curve (left) and model calibration plot (right) for the machine learning model in the full longitudinal replication cohort.

Fig. 2.

Area under the ROC curve (AUC) and 95% confidence intervals within clinical and demographic strata of the cohort compared to the overall full cohort AUC (horizontal dotted line).

Patients with higher risk scores were enriched for eventual delirium diagnosis: the top decile of risk scores included 27.7% of the total delirium cases; if all patients were ranked from highest risk to lowest risk the 10% of patients with the highest risk scores at admission ultimately accounted for 27.7% of delirium cases by discharge. The model produced a lift of 2.4 in the highest risk quintile which captured 48.5% of all delirium cases with a delirium occurrence rate of 34%. The lowest risk quintile included only 25 delirium cases for a case rate of 2.4% and lift of 0.17. At the previously identified optimal predicted risk cut point of 0.15 – that is treating those with a predicted risk greater than 0.15 as “positive” cases and those less than the cut off as “negative” cases – the predictions produced a sensitivity of 0.62 (0.58–0.65), specificity of 0.75 (0.73–0.76), negative predictive value of 0.92 (0.91–0.93), and positive predictive value of 0.28 (0.26–0.31) relative to the eventual diagnosis of delirium. Beyond this specific cut off point for determination of high-risk patients, decision curve analysis (Fig. 3 ) showed a wide range of risk thresholds over which risk stratification by the predictive model produced superior results to either the naive allocation strategy of universal intervention (i.e., intervening to prevent delirium in every admission) or that of never-intervention (i.e., providing no intervention regardless of risk).

Fig. 3.

Decision curve analysis of net benefit of risk stratification using the delirium predictive model as compared to intervening in all patients or intervening in no patients strategies.

4. Discussion

This replication of a previously developed, externally validated delirium prediction model based on EHR data available at the time of hospital admission showed comparable discrimination and diminished calibration in this cohort of more than 5000 COVID-19 patients. While the model was not applied clinically during this period, the present evaluation reflects what would have occurred during a silent evaluative deployment – i.e., no changes were made in the model features, all of which are available at time of hospital admission, and no patients were excluded. The AUC in this replication cohort of subsequent waves of patients with COVID-19 was 0.75, similar to that previously reported for this model in initial external validation cohort patients with COVID-19 during the initial wave of cases and more broadly consistent with those of diverse delirium prediction models (independent of COVID-19) summarized in a recent systematic review of the topic [64,68]. Similarly, this wide range of results falls within the range of results seen for predictions made in patients with COVID-19 about outcomes other than delirium [97]; however, the most directly comparable application of these methods to outcomes other than delirium in patients with COVID-19 produced superior discrimination to that seen in this replication cohort of delirium as an outcome [98]. This stability over subsequent waves of patients with COVID-19 is reassuring evidence of model robustness given the rapidly evolving patient population and treatment norms over the course of consecutive surges in patient volume [ 74,[99], [100], [101], [102]]. Model calibration – how closely the observed probability of the outcome matches the predicted probability of the outcome [103,104] – declined in this longitudinal validation cohort relative to that previously reported in the initial cross sectional external validation cohort. Nevertheless, patients with high predicted risk were highly enriched for actual observed cases with 27.7% of cases occurring in the highest decile of predicted risk scores. In light of emerging concern for risk of bias in machine learning models, it is notable that the overall sample AUC fell within or below the confidence interval for AUCs stratified by both race and gender [105]; however, clinical characteristics such as older age, dementia history, need for critical care, and treatment at a community hospital were associated with reduced discriminative accuracy.

The longitudinal reduction in calibration we observe is consistent with the change in observed delirium rate from 18.9% of patients in whom this model was developed to 14.0% in this subsequent replication cohort. Although the initial cohort from the spring of 2020 surge in whom the model was developed and this subsequent longitudinal replication cohort had similar average ages (62.9 vs 62.7 years respectively) the prevalence of preexisting dementia diagnosis dropped from 11.3% in the initial surge of patients to 5.6% in this replication cohort as did the proportion of patients requiring critical care, which dropped from 22.5% in the initial surge to 15.6% in this longitudinal cohort. This reduction in critical care is consistent with the declining rates of mechanical ventilation and fluctuating mortality reported elsewhere [73,99]. It is possible that the reduction in dementia diagnostic history represents a shift from nursing home residents to community dwelling older adults over subsequent waves [106,107]. These underlying changes in patient characteristics are an instance of concept drift and demonstrate the associated compromise of internal validity consistent with a rapidly evolving pandemic [ [108], [109], [110], [111]]. That model discrimination held despite this drift is reassuring; however, the impact on calibration highlights the importance of automated methods for continuous model validation and potential approaches to recalibration [112,113]. In sum, this model could be used to support population risk stratification to direct a scarce resource (e.g., delirium prevention programs) toward those patients who are most likely to be diagnosed with delirium. Whether risk predictions made by this model are adequately calibrated for counseling of individual patients about their specific predicted risk would be context specific but is of secondary relevance to the question of optimal allocation of a limited resources in the setting of pandemic associated scarcity.

This result must be viewed in the context of its strengths and weaknesses. The reliance on EHR data is a strength in that it facilitates deployment to clinical practice without expectation of any change in routine operations or real time clinical data generation; however, the use of an EHR covering an open system means that care rendered elsewhere, as in another health system, cannot be considered. The absence of relevant prior data – for example, dementia diagnosis made elsewhere – likely bias prediction results toward the null, as relevant clinical detail that would indicate increased risk may not be available. Similarly, as delirium is underdiagnosed in routine practice [[114], [115], [116], [117]], the use of a clinical diagnosis instead of active bedside screening for model evaluation likely biases results toward more conservative estimates of discrimination. The impact of this discordance between delirium diagnosis and the syndrome of delirium may not be uniformly distributed over delirium motoric subtypes in this COVID-19 population [118,119]. Outside of pandemic illness where contact and resource scarcity present unique constraints, it is possible that predictive approaches of this kind might be used to direct delirium assessments to improve on the current problem of under recognition [120]. While granting that delirium is underdiagnosed in claims data, potentially by an order of magnitude relative to formal assessment studies, secular trends in delirium diagnosis suggest increasing rates of recognition in routine practice. The 14% occurrence rate here is within the range of rates observed in studies of delirium in COVID [7,8,58,114,121]. Whereas previous research using free text identified more evidence of delirium than reliance on coded diagnosis alone that was not the case in this analysis in which sensitivity checks of our primary coded diagnosis outcome using a previously described outcome which included discharge summary text was not substantially changed and thus is not shown [57]. Given this uncertainty around the quality of coded diagnosis in COVID-19 patients during the pandemic the role and relevance of otherwise common underdiagnosis is an important area of uncertainty for future work. In addition to uncertainty about the rate of underdiagnosis in the EHR this study does not address questions about post hospital outcomes of individual patients with COVID-19 as it is longitudinal with respect to timing of cohorts not individual clinical course. The stratified analysis by dementia presents unique challenges as well as dementia, like delirium, is often underdiagnosed [[122], [123], [124], [125]] and when present delirium superimposed on dementia presents an ambiguous clinical picture [81,126].

A key strength of this work is that it is a longitudinal replication using an unmodified previously published predictive model and thus it is protected against overfitting and associated threats to replication [[127], [128], [129]]. This replication provides important information on the likely robustness of predictive models developed in the early phases of a pandemic for application in later phases of a pandemic illness as would be required in any attempt to do algorithmic allocation [130]; however, it does so in the setting of a single geographic region, thus this result represents a replication over time but not a replication over place. Nevertheless, complete replications of this sort are rare and an important assessment of model robustness [131]. Additionally, this model makes a single risk prediction at the moment of inpatient admission about risk of delirium diagnosis over the course of the subsequent hospitalization – this is both a strength and a limitation. The strength of this setup is that the prediction is based on facts which were already available in the EHR at the time of admission and thus the reported predictions could be made in the operational clinical EHR and doing so would be well-timed to stratify delirium preventative resources; on the other hand, the limitation of this prediction setup is that it is incapable of targeting the specific moment – over a potentially lengthy hospitalization – at which the fluctuation course of delirium would produce the greatest symptom burden. Delirium has been formulated in a diathesis stress model. Within that frame, this pre-admission risk prediction is attending primarily to the diathesis [132]. Finally, it is important to note that this model of all patients with COVID-19 did not achieve equivalent discrimination to that produced by models developed for targeted clinical context like surgical and critical care [133,134]. This difference within clinical populations is of particular importance as improvements at the upper limit of AUC are more challenging [135].

4.1. Conclusions

This evaluation of a previously developed and validated machine learning model predicting delirium risk at the time of hospital admission with COVID-19 demonstrates accurate risk stratification through stable model discrimination over the course of the second surge of COVID-19 patients, in spite of substantial changes in the underlying patient characteristics. This stability of model discrimination provides evidence that tools of this sort could be developed in the initial phases of a pandemic and applied thereafter; however, close monitoring would be required to assess model performance over subsequent waves in the setting of changing treatment patterns or patient cohorts. Nevertheless, it is reassuring that the performance of this model was stable despite changes in the patient population and treatment approaches over the months following initial model development. Beyond the pandemic setting, this longitudinal replication provides evidence that predictive models applied to EHR data on admission could support optimal allocation of scarce resources like prevention efforts or formal diagnostic efforts to improve recognition.

The following are the supplementary data related to this article.

Supplemental Table 1

Logistic regression predictive model coefficients of previously developed and validated model which were used without modification – refitting, calibrating, or otherwise – in this longitudinal replication.

Supplemental Table 2

Area under the ROC curve (AUC) and 95% confidence intervals by clinical and demographic strata of the cohort for comparison to the overall full cohort AUC as shown in Fig. 2.

Data availability statement

The IRB approval under which this individual health information was used does not allow redistribution of the clinical data.

Disclosures

Mr. Castro and Ms. Hart report no conflict of interest. Dr. Sacks has received research funding from the Carney Family Foundation. Dr. Perlis has received consulting fees from Burrage Capital, Genomind, RID Ventures, and Takeda. He holds equity in Outermost Therapeutics and Psy Therapeutics. Dr. McCoy has received research funding from the Brain and Behavior Research Foundation, National Institute of Mental Health, National Institute of Nursing Research, National Human Genome Research Institute Home, and Telefonica Alfa.

Data availability

The authors do not have permission to share data.