Skip to main content
PMC home page
JAMA Network logoLink to JAMA Network
. 2019 Jul 12;2(7):e196972. doi: 10.1001/jamanetworkopen.2019.6972

Development and Validation of a Deep Learning Algorithm for Mortality Prediction in Selecting Patients With Dementia for Earlier Palliative Care Interventions

Liqin Wang 1,2,✉, Long Sha 3, Joshua R Lakin 1,4,5, Julie Bynum 6, David W Bates 1,2, Pengyu Hong 3, Li Zhou 1,2
PMCID: PMC6628612  PMID: 31298717

Key Points

Question

How does a deep learning algorithm using patient demographic information and longitudinal clinical notes to predict mortality risk perform as a proxy indicator for identifying patients with dementia who need palliative care?

Findings

In this cohort study, for a validation data set of 2692 adult patients with dementia, mortality prediction models reached an area under the receiver operating characteristic curve of 0.978 for predicting death in 6 months, 0.956 for 1 year, and 0.943 for 2 years.

Meaning

Deep learning appears to show promising results in mortality risk stratification in patients with dementia.

This cohort study develops and validates a deep learning algorithm using longitudinal electronic health records to predict mortality risk as a proxy indicator for identifying patients with dementia who may benefit from palliative care.

Abstract

Importance

Early palliative care interventions drive high-value care but currently are underused. Health care professionals face challenges in identifying patients who may benefit from palliative care.

Objective

To develop a deep learning algorithm using longitudinal electronic health records to predict mortality risk as a proxy indicator for identifying patients with dementia who may benefit from palliative care.

Design, Setting, and Participants

In this retrospective cohort study, 6-month, 1-year, and 2-year mortality prediction models with recurrent neural networks used patient demographic information and topics generated from clinical notes within Partners HealthCare System, an integrated health care delivery system in Boston, Massachusetts. This study included 26 921 adult patients with dementia who visited the health care system from January 1, 2011, through December 31, 2017. The models were trained using a data set of 24 229 patients and validated using another data set of 2692 patients. Data were analyzed from September 18, 2018, to May 15, 2019.

Main Outcomes and Measures

The area under the receiver operating characteristic curve (AUC) for 6-month and 1- and 2-year mortality prediction models and the factors contributing to the predictions.

Results

The study cohort included 26 921 patients (16 263 women [60.4%]; mean [SD] age, 74.6 [13.5] years). For the 24 229 patients in the training data set, mean (SD) age was 74.8 (13.2) years and 14 632 (60.4%) were women. For the 2692 patients in the validation data set, mean (SD) age was 75.0 (12.6) years and 1631 (60.6%) were women. The 6-month model reached an AUC of 0.978 (95% CI, 0.977-0.978); the 1-year model, 0.956 (95% CI, 0.955-0.956); and the 2-year model, 0.943 (95% CI, 0.942-0.944). The top-ranked latent topics associated with 6-month and 1- and 2-year mortality in patients with dementia include palliative and end-of-life care, cognitive function, delirium, testing of cholesterol levels, cancer, pain, use of health care services, arthritis, nutritional status, skin care, family meeting, shock, respiratory failure, and swallowing function.

Conclusions and Relevance

A deep learning algorithm based on patient demographic information and longitudinal clinical notes appeared to show promising results in predicting mortality among patients with dementia in different time frames. Further research is necessary to determine the feasibility of applying this algorithm in clinical settings for identifying unmet palliative care needs earlier.

Introduction

A growing number of US adults have Alzheimer disease and related dementias (ADRD).1,2,3 As dementia progresses, patients frequently receive interventions that can add to this burden,4,5 including tube feeding6,7 and hospital transfers.8 These treatments, if unhelpful in achieving patient and family goals, can potentially contribute to poor quality of life and family dissatisfaction,9 while also driving higher health care expenditures at the end of life.10,11,12 Early palliative care interventions hold promise in the population with ADRD,13,14 because the delivery of palliative care improves patient care and family bereavement outcomes and results in more appropriate use of health care resources in other patient populations.15,16,17,18,19,20 As such, national organizations are intensifying calls for increasing the reach of palliative care to more patients.21,22 However, knowing which patients may benefit from palliative care and when is difficult and remains a key barrier to expanding reach. Data suggest that patients with ADRD receive palliative care late in life, possibly interfering with accrual of benefit to patients and families.23 A predictive tool improving the timeliness of palliative care interventions in patients with ADRD could help to optimally target scarce resources and improve patient care.

Current approaches to identification of patients with palliative care needs rely heavily on busy health care professionals, claims data, and logistic regression models, each of which has inherent limitations.24,25,26 Several survival prediction tools, such as the Palliative Performance Scale27 and Palliative Prognostic Score,28 have been developed for specialty palliative care or hospice applications and are based on exponential multiple regression analysis by considering expert-curated features such as functional ability, self-care, and oral intake. However, these tools are limited by requiring expert clinical opinions for each patient.27,28 Prior efforts to develop prognostic models to predict survival for larger groups of patients specifically with ADRD have been limited to specific clinical settings (eg, nursing homes29) or data sets (eg, caregiver interviews and claims records30). Although shorter-term prediction models, such as 6-month29 and 12-month30,31 predictions, are helpful for some palliative care applications (eg, hospice care), longer-term prediction models are also important in ADRD not only owing to the nature of the disease and associated cognitive and functional decline but also because many of the essential requirements of high-value palliative care, such as advance care planning, serious illness communications, and meaningful conversations about patients’ goals and values, must be performed earlier in the disease course.

To identify patients with ADRD who may benefit from earlier palliative care interventions, we developed and validated 6-month, 1-year, and 2-year mortality prediction models, with a primary focus on the 2-year model, using a deep neural network and longitudinal clinical notes from electronic health records (EHRs). We also improved the transparency and interpretability of complex machine learning predictive models by determining the predictive factors derived from clinical notes associated with mortality in dementia populations.

Methods

Clinical Setting and Data Sources

This retrospective cohort study was conducted at Partners HealthCare System (PHS), a nonprofit integrated health care system in Boston, Massachusetts. The PHS care delivery network was founded by 2 academic medical centers (Brigham and Women’s Hospital and Massachusetts General Hospital) and includes multiple community hospitals, specialty facilities, community health centers, and other health-related entities (such as a rehabilitation hospital). We collected data from the PHS Research Patient Data Registry, a clinical data registry that gathers medical records from various hospital systems, and the Enterprise Data Warehouse (EDW), which stores patients’ EHR data. The Massachusetts Death Index was obtained to supplement death data available in PHS data sets. This study was approved by the institutional review board of PHS with waiver of informed consent from study participants for secondary use of electronic health records. This study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline for cohort studies.

Study Cohort

We identified a cohort of study patients older than 18 years with ADRD who visited PHS from January 1, 2011, through December 31, 2017, using International Classification of Diseases, Ninth Revision, Clinical Modification codes 290, 294.1, 294.2, 331.0, 331.1, 331.2, and 331.82 and International Statistical Classification of Diseases, Tenth Revision, Clinical Modification codes F00 to F03, G30.0, G30.1, G30.8, G30.9, G31.0, G31.1, G31.83, and G31.9 (see the fully expanded list of codes and their descriptions in eTable 1 in the Supplement). We further restricted the cohort to those patients who (1) were known to be deceased or had clinical notes available within 2 years before the date of their last visit in our health care system and (2) had more than 1 documented clinical note.

Data Preparation

In this study, we included patient age, sex, race, ethnicity, educational level, and marital status, all of which were reported by participants or proxy respondents (such as a family caregiver) and collected in the EHR as part of regular clinical care. We obtained all types of clinical notes (eg, clinic visit notes, discharge summaries, and consultation notes) except narrative clinical reports (eg, radiology and pathology reports) documented during the study period. We further aggregated clinical notes by date, concatenating all notes documented on the same date into 1 note event in the data set. Thereafter, we used a natural language processing approach similar to the one developed in a previous study23 and generated 500 latent topics from clinical notes as well as topic document proportion scores (indicating the proportion of a document containing information about the topic) for each note event.

To prepare a labeled data set of the study cohort for predictive modeling, we obtained the vital status (alive or deceased) of the study cohort from Partners’ Enterprise Data Warehouse and the Massachusetts Death Index. We linked our study cohort to the Massachusetts Death Index to obtain additional death information by exact matching on a combination of Social Security number, sex, and date of birth or a combination of patient name, sex, date of birth, and city of residence. We also retrieved patients’ date of last visit to a PHS-affiliated health care facility as of September 18, 2018, to determine the vital status of the study cohort.

Development of the Models

We formulated our mortality prediction as a classification task in which the model aimed to make binary predictions at a time of a specific note event, namely whether the patient was going to die in 6 months or 1 or 2 years. Deep learning is a process of training deep neural networks to perform such a classification task. We chose the long short-term memory (LSTM) network,32 a novel recurrent neural network in conjunction with an appropriate gradient-based learning algorithm, as the network architecture in our algorithm, given LSTM’s ability to model longitudinal EHR data. Two prior studies33,34 have used LSTM networks for predicting in-hospital mortality, postdischarge mortality, and 30-day readmissions. The deep learning neural network we constructed is composed of 2 stacked LSTM layers with 2 attention layers: one placed between the input layer and the LSTM layers and the other between the LSTM layers and the output layers (Figure 1). The inputs of our deep neural network model were the variables concatenated with topic document proportion scores of 500 topics and patient demographic variables. The stacked LSTM layers supported a hierarchical abstraction of the input data. The attention layers were used to improve model performance as well as trace the importance of temporal inputs while the model was making predictions.35 Thus, we were able to extract a weight for each feature at a specific note event representing its importance to the model’s prediction. These weights were used to rank the predictive power of the topic features with respect to the 6-month, 1-year, and 2-year mortality predictions.

Figure 1. Overview of the Predictive Modeling Using Longitudinal Clinical Notes and Demographics of Patients With Dementia.

Figure 1.

Open in a new tab

A, Natural language processing (NLP) preprocessing to convert raw clinical text into a bag of words after removing punctuation and other symbols. D indicates document. B, Generation of the topic (T) features from clinical documents (D) using topic modeling. C, Concatenation of the topic features and the demographic features to form input to the neural network. D, A stacked long short-term memory (LSTM) neural network with 2 attention layers (boxes marked in gray). X indicates input variables of the neural network, which were also the results from step C.

We randomly split the study cohort into development and validation data sets with a ratio of 9:1. The development data set was further divided into training and tuning parts, with 89% of the data set used to optimize the weights of the neural network according to cross-entropy loss function and 11% used to optimize hyperparameters (eg, learning rates, depth of the network, size of the hidden layers). We set the minimum required number of note events to 2 so that our model would have a sufficient medical history for each patient to make reliable predictions. Any patients having fewer than 2 note events were excluded from the training and validation data sets. We tuned other hyperparameters using a grid search for achieving an optimal performance in the tuning data set. In addition, we used dropout to avoid model overfitting, a method that shuts down a random percentage of artificial neurons during each training epoch to reduce interdependent learning among the neurons in the model and to force the model to learn more robust internal representations.

Statistical Analysis

Data were analyzed from September 18, 2018, to May 15, 2019. We validated the final 6-month, 1-year, and 2-year mortality prediction models using the validation data set and reported the performance of our approach using the area under the receiver operating characteristics curve (AUC). The 95% CIs were computed with 2000 stratified bootstrap replicates.36 All statistical analyses were performed using R software, version 3.5.3 (R Foundation for Statistical Computing).37

Results

We present patient demographic information and note event characteristics in Table 1. The study cohort included 26 921 patients (16 263 women [60.4%] and 10 658 men [39.6%] men; mean [SD] age, 74.6 [13.5] years) who met the inclusion criteria. Of those patients, we reserved 10.0% (n = 2692) for validation, leaving the remainder (n = 24 229) for the development of the models. Of the 24 229 patients in the development set, 14 632 (60.4%) were women and 9597 (39.6%) were men, and the mean (SD) age was 74.8 (13.2) years. Of the 2692 patients in the validation set, 1631 (60.6%) were women and 1061 (39.4%) were men, and the mean (SD) age was 75.0 (12.6) years. Among these 2 data sets, 23 039 patients were white (85.6%), 24 661 were non-Hispanic (91.6%), and 12 385 (46.0%) died from January 1, 2011, through September 18, 2018. We labeled a total of 959 628 note events with 6-month, 1-year, and 2-year mortality. A mean of 35 to 36 note events were found per person. More than 75% of the note events were documented more than 2 years before death or last patient visit to PHS facilities. In addition, more notes were documented nearer to death (eg, 47 219 note events 0-3 months before death vs 28 472 note events 4-6 months before death) in the development data set.

Table 1. Characteristics of the Study Cohort and Note Events.

Characteristic Data Seta
Development (n = 24 229) Validation (n = 2692)
Age, mean (SD), yb 74.8 (13.2) 75.0 (12.6)
Diedc 11 138 (46.0) 1247 (46.3)
Female sex 14 628 (60.4) 1631 (60.6)
Race
White 20 734 (85.6) 2305 (85.6)
Black 1302 (5.4) 159 (5.9)
Others 515 (2.1) 47 (1.7)
Unknown 1678 (6.9) 182 (6.8)
Ethnicity
Non-Hispanic 22 190 (91.6) 2471 (91.8)
Hispanic 1400 (5.8) 147 (5.5)
Unknown 639 (2.6) 74 (2.7)
Marital status
Married or partnered 10 490 (43.3) 1162 (43.2)
Single, divorced, or widowed 12 324 (50.9) 1378 (51.2)
Unknown 1415 (5.8) 152 (5.6)
Educational level
College and above 6955 (28.7) 755 (28.0)
High school or equivalent 7392 (30.5) 812 (30.2)
Did not complete high school 2181 (9.0) 239 (8.9)
Unknown 7701 (31.8) 886 (32.9)
No. of total note eventsd 863 160 96 468
No. of note events per patient, mean (SD) 35.6 (49.2) 38.8 (49.8)
No. of note events in time before death, mo
0-3 47 219 (5.5) 5238 (5.4)
4-6 28 472 (3.3) 3223 (3.3)
7-12 49 709 (5.8) 5684 (5.9)
13-24 82 767 (9.6) 9465 (9.8)
≥25e 654 993 (75.9) 72 858 (75.5)
Open in a new tab
a

Unless otherwise indicated, data are expressed as number (percentage) of patients. Percentages have been rounded and may not total 100.

b

Calculated at the beginning of the study period (ie, January 1, 2011).

c

Collected from January 1, 2011, through September 18, 2018.

d

The note events met the following inclusion criteria: (1) can be labeled in terms of 2-year mortality and (2) have more than 10 words after the natural language processing preprocessing.

e

A significant increase of note events documented more than 2 years before death was due to the inclusion of patients who were still living as of the most recent date of encounter recorded in the patient’s record in our health care system.

The trained models made predictions at the time stamps of all the note events of the patients in the validation data set. By checking the classification of each prediction event against patient vital status, our proposed model reached an AUC of 0.943 (95% CI, 0.942-0.944) for predicting 2-year mortality, an AUC of 0.956 (95% CI, 0.955-0.956) for predicting 1-year mortality, and an AUC of 0.978 (95% CI, 0.977-0.978) for predicting 6-month mortality (Figure 2).

Figure 2. Receiver Operating Characteristic Curves of the Deep Learning Models in Predicting Patient Mortality.

Figure 2.

Open in a new tab

In a validation data set of 2692 patients with Alzheimer disease and related dementia, the deep learning–based models showed high note events–level classification of 6-month, 1-year, and 2-year mortality, achieving areas under the receiver operating characteristic curve of 0.978 (95% CI, 0.977-0.978) for the 6-month model, 0.956 (95% CI, 0.955-0.956) for the 1-year model, and 0.943 (95% CI, 0.942-0.944) for the 2-year model.

At the patient level, the weights of the topics extracted from the attention layer during the prediction were informative of which topics at which prior note event were predictive (Figure 3). By summing the weights of each topic in making predictions at all note events of the 2692 patients in the validation cohort, we identified a list of topics as top-ranked predictive features for 6-month, 1-year, and 2-year mortality. The top-ranked latent topics associated with 6-month and 1- and 2-year mortality in patients with dementia include palliative and end-of-life care, cognitive function, delirium, testing of cholesterol levels, cancer, pain, use of health care services, arthritis, nutritional status, skin care, family meeting, shock, respiratory failure, and swallowing function. Table 2 shows the top 20 ranked topics associated with 6-month, 1-year, and 2-year mortality as well as their labels annotated by one of us (J.R.L.); the top 100 ranked topics are available in eTables 2 to 4 in the Supplement.

Figure 3. Topic Attention Heatmap and Corresponding Note Events Predicting 2-Year Mortality.

Figure 3.

Open in a new tab

A, Topic attention heatmap showing, in predicting 2-year mortality at the time stamp of the last note event, the contribution of selected 20 predictive topics from prior 50 note events. B, The topic numbers and their top 10 probable words. BUN indicates blood urea nitrogen; CHF, congestive heart failure; Cre, creatinine; DNI, do not intubate; DNR, do not resuscitate; Glu, glucose; Hct, hematocrit; Hgb, hemoglobin; ICMP, intensive care management program; Plt, platelet; PRN, prescription as needed; SNF, skilled nursing facility; VNA, Visiting Nurse Association; and WBC, white blood count.

Table 2. Top 20 Predictive Topics Associated With 6-Month, 1-Year, and 2-Year Mortality.

Rank Manual Label Top 15 Probable Words
Top-Ranked Predictive Topics for 2-Year Model
1 Palliative and end-of-life care Care hospice family comfort palliative DNI DNR PRN goal morphine CMO discussion dementia measure pain
2 Cognitive function Status dementia unable mental baseline eye command nurse verbal alter hypernatremia open due poor lethargy
3 Cholesterol level testing Cholesterol LDL result test total blood compare HDL bad normal function triglyceride good hemoglobin medical
4 Delirium Agitation agitate dementia Seroquel delirium Haldol continue Zyprexa PRN sitter behavior psych time medication trazodone
5 Laboratory testing Range normal detail test blood result function check glucose creatinine potassium kidney total calcium BUN
6 Cancer Cancer lung metastatic disease chemotherapy oncology cycle radiation cell show chemotherapy tumor carcinoma mass adenocarcinoma
7 Pain evaluation and treatment Pain therapy time treatment week increase level exercise physical functional report activity tissue visit hip
8 Hospital care Date information case phone admit information referral status hospital Salem care bed gender contact page
9 Results communication Result test letter question dear receive contact manager share normal hesitate blood function show report
10 Facility care Facility live assist nurse dementia rehabilitation SNF care transfer discharge staff term fall ALF long
11 Nutritional status Nutrition intake weight protein day supplement goal meal clinical calorie kcal daily Ensure diet continue
12 Spanish documentationa Los para una con usted por tiene puede sus del medico dolor como medicamentos sobre
13 Health care encounter Hospital general medication Massachusetts management medicine associate fax internal phone pharmacy electronically transmit prescription prepare
14 Arthritis Knee pain leave joint osteoarthritis injection replacement medial surgery arthritis effusion total lateral bilateral motion
15 Mental status examination Disorder Axis problem visit mood anxiety coherent risk direct office treatment current pain exam status
16 Heart failure Lasix CHF edema weight volume overload increase diuresis home daily heart failure SOB admission fluid
17 Skin care Skin lesion include extremity sun scalp discuss back perform exam papule upper dermatology face nevus
18 Family meeting Family care discuss meet time discussion make plan team decision son discus understand medical risk
19 General medical care Care pain respiratory dementia extend gait edema weight gastrointestinal musculoskeletal med review wheeze clear erythema
20 Pain evaluation and treatment Pain back leg Tylenol leave neck low day lower side muscle tenderness week worse ibuprofen
Top-Ranked Predictive Topics for 1-Year Model
1 Palliative and end-of-life care Care hospice family comfort palliative DNI DNR PRN goal morphine CMO discussion dementia measure pain
2 Cognitive function Status dementia unable mental baseline eye command nurse verbal alter hypernatremia open due poor lethargy
3 Laboratory testing Range normal detail test blood result function check glucose creatinine potassium kidney total calcium BUN
4 Cholesterol level testingb Cholesterol LDL result test total blood compare HDL bad normal function triglyceride good hemoglobin medical
5 Results communication Result test letter question dear receive contact manager share normal hesitate blood function show report
6 Medication delivery Tablet day tablet BID capsule QHS PRN direct acid HCL unit release vitamin visit TID
7 Family meeting Family care discuss meet time discussion make plan team decision son discus understand medical risk
8 Physical examination Normal time note sit review pulse status history interpretation inspection pain physician resp skin respiratory
9 Delirium Agitation agitate dementia Seroquel delirium Haldol continue Zyprexa PRN sitter behavior psych time medication trazodone
10 Health care encounter Hospital general medication Massachusetts management medicine associate fax internal phone pharmacy electronically transmit prescription prepare
11 Facility care Facility live assist nurse dementia rehabilitation SNF care transfer discharge staff term fall ALF long
12 Cholesterol level testingb Test cholesterol blood follow function laboratory phone recent normal office dear range medicine Parkman kidney
13 Pain evaluation and treatment Pain therapy time treatment week increase level exercise physical functional report activity tissue visit hip
14 Spanish documentationa Los para una con usted por tiene puede sus del mdico dolor como medicamentos sobre
15 Hospital care Date information case phone admit information referral status hospital Salem care bed gender contact page
16 Nursing care Continue progress rate output intake hour today SPO urine monitor total overnight shift nurse event
17 Mental status examination Disorder axis problem visit mood anxiety coherent risk direct office treatment current pain exam status
18 Swallowing function Liquid swallow diet aspiration dysphagia thick puree SLP nectar solid thin speech continue soft consistency
19 Cancer Cancer lung metastatic disease chemotherapy oncology cycle radiation cell show chemotherapy tumor carcinoma mass adenocarcinoma
20 Nutritional status Nutrition intake weight protein day supplement goal meal clinical calorie kcal daily ensure diet continue
Top-Ranked Predictive Topics for 6-Month Model
1 Palliative and end-of-life care Care hospice family comfort palliative DNI DNR PRN goal morphine CMO discussion dementia measure pain
2 Cognitive function Status dementia unable mental baseline eye command nurse verbal alter hypernatremia open due poor lethargy
3 Laboratory testing Range normal detail test blood result function check glucose creatinine potassium kidney total calcium BUN
4 Cholesterol level testing Cholesterol LDL result test total blood compare HDL bad normal function triglyceride good hemoglobin medical
5 Results communication Result test letter question dear receive contact manager share normal hesitate blood function show report
6 Physical examination Normal time note sit review pulse status history interpretation inspection pain physician resp skin respiratory
7 Family meeting Family care discuss meet time discussion make plan team decision son discuss understand medical risk
8 Healthcare encounter Hospital general medication Massachusetts management medicine associate fax internal phone pharmacy electronically transmit prescription prepare
9 Delirium Agitation agitate dementia Seroquel delirium Haldol continue Zyprexa PRN sitter behavior psych time medication trazodone
10 Nutritional status Nutrition intake weight protein day supplement goal meal clinical calorie kcal daily Ensure diet continue
11 Hospital care Progress hospitalization adult absence risk continue pediatric fall actual sign discharge infection symptom condition pressure
12 Respiratory failure Respiratory BIPAP failure pulmonary oxygen hypoxia sit ICU edema status transfer Lasix distress require improve
13 Shock Shock sepsis transfer ICU MICU hypotension septic failure set pressor continue fluid require improve respiratory
14 Swallowing function Swallow SLP liquid aspiration oral speech thin dysphagia solid diet puree language cough consistency thick
15 Swallowing function Liquid swallow diet aspiration dysphagia thick puree SLP nectar solid thin speech continue soft consistency
16 Physical examination Pressure blood normal edema pulse weight clear chest murmur today year regular daily heart extremity
17 Medication delivery Tablet day tablet BID capsule QHS PRN direct acid HCL unit release vitamin visit TID
18 Intensive care Intubate vent airway goal day continue tube Fio CMH respiratory care line rate ICU ETT
19 Facility care Facility live assist nurse dementia rehabilitation SNF care transfer discharge staff term fall ALF long
20 General medical care Care pain respiratory dementia extend gait edema weight gastrointestinal musculoskeletal med review wheeze clear erythema
Open in a new tab

Abbreviations (approximated from clinical notes): ALF, assisted living facility; BID, twice daily; BIPAP, bilevel positive airway pressure; BUN, blood urea nitrogen; CHF, congestive heart failure; CMH, centimeters of water; CMO, comfort measures only; DNI, do not intubate; DNR, do not resuscitate; ETT, endotracheal tube; Fio, fraction of inhaled oxygen; HCL, hydrochloride; HDL, high-density lipoprotein; ICU, intensive care unit; kcal, kilocalorie; LDL, low-density lipoprotein; MICU, medical ICU; PRN, prescription as needed; psych, psychology or psychiatry or some variation on these terms; QHS, at bedtime; resp, respiratory or respiration or some similar variations; SLP, speech-language pathology; SNF, skilled nursing facility; SOB, shortness of breath; SPO, peripheral capillary oxygen saturation; TID, three times daily.

a

This topic groups common words in Spanish because of the inclusion of clinical notes written in Spanish, primarily among the notes for communication with patients, including patient letters and instructions.

b

Topics with similar words were labeled with the same name.

Discussion

This study demonstrates that a deep neural network trained using a large data set with patient demographics and longitudinal clinical notes from the EHR can be accurate and useful in predicting 6-month, 1-year, and 2-year mortality and thus could be used as a proxy for selecting patients who may benefit from palliative care assessment. The high performance (AUC scores) of all 3 models shows that clinical notes along with patient demographics are informative, and the deep learning neural network structure can successfully capture short- and long-range longitudinal patterns. In addition, converting clinical notes into clinically meaningful topics using topic modeling allows us to trace and visualize how the model made its prediction for each patient (Figure 3). Meanwhile, at the population level, the model helps us to identify what factors are strongly associated with mortality risk in different time frames in patients with ADRD (Table 2).

In the past, studies of mortality prediction have relied on claims data,38 administrative data,39 or other types of data (eg, surveys),40 but few have used clinical notes. We believe that this study is the first to investigate clinical notes in a deep neural network to identify topics associated with mortality prediction among patients with ADRD. In the LSTM-based neural network, clinical notes contribute to mortality prediction in 2 aspects: the longitudinal patterns of the documentation and the content of clinical notes. First, frequent documentation in the medical record likely indicates increasing severity of illness and worsening frailty in the context of ADRD; thus, with the help of the LSTM neural network, long- and short-term longitudinal patterns can be identified for mortality prediction. Second, topics generated using the topic modeling method captured semantic and syntactic structures of large quantities of clinical notes, providing rich information for mortality prediction. Among 500 topics, top-ranked predictive factors associated with 6-month, 1-year, and 2-year mortality include palliative and end-of-life care, cognitive function (eg, dementia status, delirium), laboratory testing (eg, testing of cholesterol levels), cancer, pain, use of health care services (eg, hospital or facility care, health care encounter, intensive care, and nursing care), arthritis, nutritional status, skin care, family meeting, result communication, swallowing function, shock, respiratory failure, and medication delivery, among others. Some of these topics indicate that health care professionals may recognize a patient’s decline (such as notation of palliative and end-of-life care), while others may signal changing patient conditions that health care professionals have yet to recognize (such as cognitive function, delirium, and functional status). Only a few studies explicitly list variables used to predict mortality in the ADRD population. For example, Mitchell et al29,41 included length of stay, dyspnea, pressure ulcers, total functional dependence, being bedbound most of the day, insufficient intake, bowel incontinence, body mass index, weight loss, and congestive heart failure as variables that best predict 6-month survival. Using the topic modeling method, we were able to capture topics that seem similar to variables selected a priori as well as additional variables that may not be available in many structured data.

Our models can be calculated with much less time and effort in large patient populations compared with existing screening methods (eg, the “surprise question” method).24,42 Previous studies demonstrated that health care professionals, although directionally generally correct, have trouble estimating the timing of death.43,44 Long-term predictions are generally more difficult for humans; this may also apply to the machine, because our mortality prediction models achieved slightly lower performance when the prediction time frames became longer. However, our 2-year model still reached a high AUC of 0.943. Therefore, using deep learning predictive models in patient stratification in clinical practice has notable promise for identifying patients with ADRD who are approaching their last 1 or 2 years of life.

Although mortality is not the only important factor contributing to assessment of need for palliative care, tools such as this algorithm may provide an important proxy that health care professionals and systems can use to consider patients for possible palliative care interventions. By adjusting the sensitivity and specificity along the receiver operating characteristic curve, deep learning–based tools may be used to decrease the burden on health care professionals by identifying a manageable denominator of patients for consideration for interventions according to available palliative care resources. They may also help guide prioritization of patients’ needs based on predicted probability of mortality in a certain time frame. Importantly, these models should not be used in the absence of input from health care professionals, because computer-predicted mortality alone is not a decisive indicator of palliative care needs; the benefit of palliative care depends on far more than risk of death (eg, individual preferences, functional and quality of life effects of serious illness, psychosocial and spiritual needs, and burden of illness on caregiving networks). In addition, we have chosen a longer time frame of 2 years to target driving earlier conversations about patients’ goals and values in ADRD and to focus on patient-centric conversations rather than system-centric decisions such as enrollment in hospice.

Limitations

One major limitation of our study is that our models have not been validated using external data sets. Because of population diversity and clinical documentation variations among different health care systems,45 we suspect that models trained from the data of one health care system may require additional tuning to be adaptable to other systems or EHRs. Therefore, until a systematic validation is performed, including using a different data set to assess the models’ generalizability, these models should not be widely applied to other health care systems. Second, machine learning–based models developed using EHR data may be subject to bias because the EHR generally contains more medical information for sicker patients, and this decreases the generalizability of the models to non-EHR settings.46,47 Third, model-based screening can only make predictions at the times when notes are available for patients, requiring a minimum of 2 note events. This limitation may affect the model’s capability to make predictions for all patients with ADRD at any time. Fourth, the ranking of the predictive topics, which was generated based on the attention of the neural network during the prediction for the validation cohort, does not directly correlate to the proportion of notes or patients to whom these topics apply, and such rankings may be subject to change as the predictive cohort changes. Fifth, prediction of mortality is only one component of identifying patients who may benefit from palliative care, and future predictive modeling efforts should move beyond mortality prediction to work on identifying broader needs for populations of seriously ill patients, such as predicting functional decline and effects on quality of life.48

Conclusions

In evaluating predictive models as proxies for identifying patients with ADRD for early palliative care interventions, a deep machine learning algorithm using patient demographic information and topics derived from longitudinal clinical notes appears to show promising results in predicting 6-month, 1-year, and 2-year mortality. Further research is necessary to determine the feasibility of applying this algorithm in the clinical setting for identifying unmet palliative care needs earlier in patients with dementia.

Supplement.

eTable 1. List of ICD Diagnosis Codes for Identifying Patients Who Have Alzheimer’s Disease and Related Dementias

eTable 2. Top 100 Ranked Predictive Topics for Predicting 2-Year Mortality

eTable 3. Top 100 Ranked Predictive Topics for Predicting 1-Year Mortality

eTable 4. Top 100 Ranked Predictive Topics for Predicting 6-Month Mortality

Click here for additional data file. (349.1KB, pdf)

References

  • 1.Alzheimer’s Association. 2018 Alzheimer’s disease facts and figures. Alzheimers Dement. 2018;14(3):-. doi: 10.1016/j.jalz.2018.02.001 [DOI] [PubMed] [Google Scholar]
  • 2.Alzheimer’s Association. 2017 Alzheimer’s disease facts and figures. Alzheimers Dement. 2017;13(4):325-373. doi: 10.1016/j.jalz.2017.02.001 [DOI] [Google Scholar]
  • 3.Alzheimer’s Association 2016 Alzheimer’s disease facts and figures. Alzheimers Dement. 2016;12(4):459-509. doi: 10.1016/j.jalz.2016.03.001 [DOI] [PubMed] [Google Scholar]
  • 4.Mitchell SL, Teno JM, Kiely DK, et al. . The clinical course of advanced dementia. N Engl J Med. 2009;361(16):1529-1538. doi: 10.1056/NEJMoa0902234 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Mitchell SL, Shaffer ML, Cohen S, Hanson LC, Habtemariam D, Volandes AE. An advance care planning video decision support tool for nursing home residents with advanced dementia: a cluster randomized clinical trial. JAMA Intern Med. 2018;178(7):961-969. doi: 10.1001/jamainternmed.2018.1506 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Mitchell SL, Mor V, Gozalo PL, Servadio JL, Teno JM. Tube feeding in US nursing home residents with advanced dementia, 2000-2014. JAMA. 2016;316(7):769-770. doi: 10.1001/jama.2016.9374 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Mitchell SL, Kiely DK, Lipsitz LA. The risk factors and impact on survival of feeding tube placement in nursing home residents with severe cognitive impairment. Arch Intern Med. 1997;157(3):327-332. doi: 10.1001/archinte.1997.00440240091014 [DOI] [PubMed] [Google Scholar]
  • 8.Gozalo P, Teno JM, Mitchell SL, et al. . End-of-life transitions among nursing home residents with cognitive issues. N Engl J Med. 2011;365(13):1212-1221. doi: 10.1056/NEJMsa1100347 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Engel SE, Kiely DK, Mitchell SL. Satisfaction with end-of-life care for nursing home residents with advanced dementia. J Am Geriatr Soc. 2006;54(10):1567-1572. doi: 10.1111/j.1532-5415.2006.00900.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Hill J, Fillit H, Shah SN, del Valle MC, Futterman R. Patterns of healthcare utilization and costs for vascular dementia in a community-dwelling population. J Alzheimers Dis. 2005;8(1):43-50. doi: 10.3233/JAD-2005-8105 [DOI] [PubMed] [Google Scholar]
  • 11.Zhao Y, Kuo TC, Weir S, Kramer MS, Ash AS. Healthcare costs and utilization for Medicare beneficiaries with Alzheimer’s. BMC Health Serv Res. 2008;8:108. doi: 10.1186/1472-6963-8-108 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Goldfeld KS, Stevenson DG, Hamel MB, Mitchell SL. Medicare expenditures among nursing home residents with advanced dementia. Arch Intern Med. 2011;171(9):824-830. doi: 10.1001/archinternmed.2010.478 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Murphy E, Froggatt K, Connolly S, et al. . Palliative care interventions in advanced dementia. Cochrane Database Syst Rev. 2016;12:CD011513. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Mitchell SL. Care of patients with advanced dementia. https://www.uptodate.com/contents/care-of-patients-with-advanced-dementia. Updated July 16, 2018. Accessed October 24, 2018.
  • 15.Wright AA, Zhang B, Ray A, et al. . Associations between end-of-life discussions, patient mental health, medical care near death, and caregiver bereavement adjustment. JAMA. 2008;300(14):1665-1673. doi: 10.1001/jama.300.14.1665 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Wendler D, Rid A. Systematic review: the effect on surrogates of making treatment decisions for others. Ann Intern Med. 2011;154(5):336-346. doi: 10.7326/0003-4819-154-5-201103010-00008 [DOI] [PubMed] [Google Scholar]
  • 17.Mack JW, Weeks JC, Wright AA, Block SD, Prigerson HG. End-of-life discussions, goal attainment, and distress at the end of life: predictors and outcomes of receipt of care consistent with preferences. J Clin Oncol. 2010;28(7):1203-1208. doi: 10.1200/JCO.2009.25.4672 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Zhang B, Wright AA, Huskamp HA, et al. . Health care costs in the last week of life: associations with end-of-life conversations. Arch Intern Med. 2009;169(5):480-488. doi: 10.1001/archinternmed.2008.587 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Morrison RS, Penrod JD, Cassel JB, et al. ; Palliative Care Leadership Centers’ Outcomes Group . Cost savings associated with US hospital palliative care consultation programs. Arch Intern Med. 2008;168(16):1783-1790. doi: 10.1001/archinte.168.16.1783 [DOI] [PubMed] [Google Scholar]
  • 20.Temel JS, Greer JA, Muzikansky A, et al. . Early palliative care for patients with metastatic non–small-cell lung cancer. N Engl J Med. 2010;363(8):733-742. doi: 10.1056/NEJMoa1000678 [DOI] [PubMed] [Google Scholar]
  • 21.Mitchell SL, Black BS, Ersek M, et al. . Advanced dementia: state of the art and priorities for the next decade. Ann Intern Med. 2012;156(1, pt 1):45-51. doi: 10.7326/0003-4819-156-1-201201030-00008 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.van der Steen JT, Radbruch L, Hertogh CM, et al. ; European Association for Palliative Care (EAPC) . White paper defining optimal palliative care in older people with dementia: a Delphi study and recommendations from the European Association for Palliative Care. Palliat Med. 2014;28(3):197-209. doi: 10.1177/0269216313493685 [DOI] [PubMed] [Google Scholar]
  • 23.Wang L, Lakin J, Riley C, Korach Z, Frain LN, Zhou L. Disease trajectories and end-of-life care for dementias: latent topic modeling and trend analysis using clinical notes. AMIA Annu Symp Proc. 2018;2018:1056-1065. [PMC free article] [PubMed] [Google Scholar]
  • 24.Lakin JR, Robinson MG, Bernacki RE, et al. . Estimating 1-year mortality for high-risk primary care patients using the “surprise” question. JAMA Intern Med. 2016;176(12):1863-1865. doi: 10.1001/jamainternmed.2016.5928 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Yourman LC, Lee SJ, Schonberg MA, Widera EW, Smith AK. Prognostic indices for older adults: a systematic review. JAMA. 2012;307(2):182-192. doi: 10.1001/jama.2011.1966 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Kelley AS, Bollens-Lund E. Identifying the population with serious illness: the “denominator” challenge. J Palliat Med. 2018;21(S2):S7-S16. doi: 10.1089/jpm.2017.0548 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Anderson F, Downing GM, Hill J, Casorso L, Lerch N. Palliative Performance Scale (PPS): a new tool. J Palliat Care. 1996;12(1):5-11. doi: 10.1177/082585979601200102 [DOI] [PubMed] [Google Scholar]
  • 28.Pirovano M, Maltoni M, Nanni O, et al. ; Italian Multicenter and Study Group on Palliative Care . A new palliative prognostic score: a first step for the staging of terminally ill cancer patients. J Pain Symptom Manage. 1999;17(4):231-239. doi: 10.1016/S0885-3924(98)00145-6 [DOI] [PubMed] [Google Scholar]
  • 29.Mitchell SL, Miller SC, Teno JM, Kiely DK, Davis RB, Shaffer ML. Prediction of 6-month survival of nursing home residents with advanced dementia using ADEPT vs hospice eligibility guidelines. JAMA. 2010;304(17):1929-1935. doi: 10.1001/jama.2010.1572 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Newcomer R, Covinsky KE, Clay T, Yaffe K. Predicting 12-month mortality for persons with dementia. J Gerontol B Psychol Sci Soc Sci. 2003;58(3):S187-S198. doi: 10.1093/geronb/58.3.S187 [DOI] [PubMed] [Google Scholar]
  • 31.Avati A, Jung K, Harman S, Downing L, Ng A, Shah NH. Improving palliative care with deep learning. BMC Med Inform Decis Mak. 2018;18(suppl 4)(suppl 4):122. doi: 10.1186/s12911-018-0677-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput. 1997;9(8):1735-1780. doi: 10.1162/neco.1997.9.8.1735 [DOI] [PubMed] [Google Scholar]
  • 33.Rajkomar A, Oren E, Chen K, et al. . Scalable and accurate deep learning with electronic health records. npj Digital Med. 2018;1(18):1-10. https://www.nature.com/articles/s41746-018-0029-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Jo Y, Lee L, Palaskar S. Combining LSTM and latent topic modeling for mortality prediction. https://arxiv.org/abs/170902842. Submitted September 8, 2017. Accessed October 24, 2018.
  • 35.Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. Paper presented at: Advances in Neural Information Processing Systems 2017. Conference; December 4, 2017; Long Beach, CA. https://papers.nips.cc/paper/7181-attention-is-all-you-need [Google Scholar]
  • 36.Carpenter J, Bithell J. Bootstrap confidence intervals: when, which, what? a practical guide for medical statisticians. Stat Med. 2000;19(9):1141-1164. doi: [DOI] [PubMed] [Google Scholar]
  • 37.Robin X, Turck N, Hainard A, et al. . pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics. 2011;12(1):77. doi: 10.1186/1471-2105-12-77 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Makar M, Ghassemi M, Cutler DM, Obermeyer Z. Short-term mortality prediction for elderly patients using Medicare claims data. Int J Mach Learn Comput. 2015;5(3):192-197. doi: 10.7763/IJMLC.2015.V5.506 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.Gupta S, Tran T, Luo W, et al. . Machine-learning prediction of cancer survival: a retrospective study using electronic administrative records and a cancer registry. BMJ Open. 2014;4(3):e004007. doi: 10.1136/bmjopen-2013-004007 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Levine ME, Crimmins EM. A comparison of methods for assessing mortality risk. Am J Hum Biol. 2014;26(6):768-776. doi: 10.1002/ajhb.22595 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Mitchell SL, Miller SC, Teno JM, Davis RB, Shaffer ML. The advanced dementia prognostic tool: a risk score to estimate survival in nursing home residents with advanced dementia. J Pain Symptom Manage. 2010;40(5):639-651. doi: 10.1016/j.jpainsymman.2010.02.014 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Downar J, Goldman R, Pinto R, Englesakis M, Adhikari NK. The “surprise question” for predicting death in seriously ill patients: a systematic review and meta-analysis. CMAJ. 2017;189(13):E484-E493. doi: 10.1503/cmaj.160775 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43.Christakis NA, Lamont EB. Extent and determinants of error in doctors’ prognoses in terminally ill patients: prospective cohort study. BMJ. 2000;320(7233):469-472. doi: 10.1136/bmj.320.7233.469 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44.Glare P, Virik K, Jones M, et al. . A systematic review of physicians’ survival predictions in terminally ill cancer patients. BMJ. 2003;327(7408):195-198. doi: 10.1136/bmj.327.7408.195 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Downing NL, Bates DW, Longhurst CA. Physician burnout in the electronic health record era: are we ignoring the real cause? Ann Intern Med. 2018;169(1):50-51. doi: 10.7326/M18-0139 [DOI] [PubMed] [Google Scholar]
  • 46.Gianfrancesco MA, Tamang S, Yazdany J, Schmajuk G. Potential biases in machine learning algorithms using electronic health record data. JAMA Intern Med. 2018;178(11):1544-1547. doi: 10.1001/jamainternmed.2018.3763 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47.Rusanov A, Weiskopf NG, Wang S, Weng C. Hidden in plain sight: bias towards sick patients when sampling patients with sufficient electronic health record data for research. BMC Med Inform Decis Mak. 2014;14:51. doi: 10.1186/1472-6947-14-51 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 48.Kelley AS. Defining “serious illness.” J Palliat Med. 2014;17(9):985. doi: 10.1089/jpm.2014.0164 [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplement.

eTable 1. List of ICD Diagnosis Codes for Identifying Patients Who Have Alzheimer’s Disease and Related Dementias

eTable 2. Top 100 Ranked Predictive Topics for Predicting 2-Year Mortality

eTable 3. Top 100 Ranked Predictive Topics for Predicting 1-Year Mortality

eTable 4. Top 100 Ranked Predictive Topics for Predicting 6-Month Mortality

Click here for additional data file. (349.1KB, pdf)

Articles from JAMA Network Open are provided here courtesy of American Medical Association

RESOURCES