Nurse Generated EHR Data Supports Post-Acute Care Referral Decision Making: Development and Validation of a Two-step Algorithm

Abstract

Objective: Build and validate a clinical decision support (CDS) algorithm for discharge decisions regarding referral for post-acute care (PAC) and to what site of care. Materials and Methods: Case studies derived from EHR data were judged by 171 interdisciplinary experts and prediction models were generated. Results: A two-step algorithm emerged with area under the curve (AUC) in validation of 91.5% (yes/no refer) and AUC 89.7% (where to refer). Discussion: CDS for discharge planning (DP) decisions may remove subjectivity, and variation in decision-making. CDS could automate the assessment process and alert clinicians of high need patients earlier in the hospital stay. Conclusion: Our team successfully built and validated a two-step algorithm to support discharge referral decision-making from EHR data. Getting patients the care and support they need may decrease readmissions and other adverse events. Further work is underway to test the effects of the CDS on patient outcomes in two hospitals.

Introduction

In response to the Readmission Reduction ⁽¹⁾ and Value Based Purchasing Programs ⁽²⁾ hospitals are increasingly collaborating with post-acute care (PAC) services to mitigate readmission risk. Between 1996 and 2010, discharge of patients to PAC settings, one of the fastest growing sectors of the Medicare program, showed a relative increase of approximately 50%.^(3,4) Partnering with and shifting care to the post-acute care setting increases the importance of properly identifying the right patients for the right settings.

Discharge planning (DP), a process undertaken during the hospital stay, is used to proactively identify patients’ needs after hospital discharge and to begin a smooth transition from hospital to home or other care settings. Yet, several issues exist that have the potential to undermine quality of care and patient safety. DP is a complex process conducted by a multidisciplinary healthcare team, (i.e., physicians, nurses, social workers, and physical therapists), in collaboration with patients and their caregivers. Older adults with multiple comorbidities, complex treatment regimens, and the rate at which they churn through the hospital system add to the complexity of decision making.^(5-7) Despite best efforts, the DP process is often rushed, with clinicians frequently not having all the necessary information or time to make optimal decisions regarding PAC referrals.⁽⁸⁾ Furthermore, studies have revealed great variability in DP assessments and models,⁽⁹⁾ clinician risk tolerance and decision-making regarding PAC referrals^(10-15) and insurance barriers and a lack of standards for post-acute care decision making during this critical time.⁽¹⁵⁾ Inconsistent practices for DP and identification of the need for PAC services can result in either over-referral for PAC services, a significant Medicare expense, or under-referral, leaving patients with unmet needs.⁽¹⁶⁾ The need for decision support tools to bring structure to this non-standardized, but critical, process is imperative.

Currently, few CDS tools exist that provide recommendations for care after hospital discharge. Barsoum and colleagues developed and validated a tool to identify patients that should not be discharged directly to home after total joint arthroplasty.⁽¹⁷⁾ Holland and team⁽¹⁸⁾ created a tool to identify high-risk patients in need of focused discharge planning, Tseng and colleagues demonstrated improved DP when using a systematic assessment tool versus traditional assessment,⁽¹⁹⁾ while others developed a tool to identify surgical patients who should not be discharged to home.⁽²⁰⁾ Our earlier work with a CDS called the Discharge Decision Support System (D2S2) identified six statistically significant factors associated with likelihood for PAC referral of older adults, including patient’s age, less than excellent self-rated health, no or intermittent help at home, major walking limitations, depression symptoms and number of comorbid conditions.⁽¹⁰⁾ The D2S2 was commercialized and is translated into practice in 38 hospitals. However, to our knowledge there are currently no CDS tools reported in the literature that provide recommendations for PAC referral for a general hospital population and considers the specific site of PAC services needed. The CDS described here meets those requirements.

Objectives

Our objective was to build and validate an expert clinical decision support system (CDS) for the discharge referral decisions of whether or not to refer patients for post-acute care and if so, to what level of care. The study aim was to define and validate the most significantly predictive model of factors to mimic nationally-based, multi-disciplinary experts’ post-acute care referral decisions. The hypotheses (H) for model building were: (H1) There will be a statistically significant correlation between the evidence-based factors (case study information) and the experts’ yes/no referral decisions. (H2) There will be a statistically significant correlation between the evidence-based factors and the site of referral. The hypotheses for the model validation were: (H3) The model will predict the expert referral yes/no decision with sensitivity, specificity and positive predictive value of greater than 80%. (H4) The model will predict the site of referral with sensitivity, specificity and positive predictive value of greater than 80%.

Methods

Setting. The data for the algorithm came from the electronic health records (EHR) of patients cared for in six hospitals located in the New England, Mid-Atlantic and Midwest regions of the United States. The hospitals ranged in size from community, regional, to a quaternary academic medical center. Selected hospitals all had a comprehensive, structured documentation system for nursing called Knowledge Based Charting (KBC) within their larger EHR (described below). The study was approved by the University Institutional Review Board (IRB) and the IRB of two hospitals that required review.

Sample. Each hospital provided retrospective, de-identified EHR data for hospitalized patients age 55 and older cared for on medical and surgical units and critical care. They excluded records coded as observation stays and admissions to skilled rehabilitation units, obstetrics and pediatrics. The study team received the files via secure file transfer. Data retrieval occurred between October 2011 and July 2012 and contained 5,333 patient records meeting our eligibility criteria. Hospitals provided between 908 to 1751 patient records each. The team completed an extensive cleaning and re-coding process, described elsewhere,⁽²²⁾ and then merged the files into one uniform dataset. To assure a nationally representative sample, the statistician drew the final study sample of 1,496 patients from the 5,333 to obtain a representative distribution of the 16 most common primary diagnoses of hospitalized patients.⁽²³⁾ The EHR data populated the electronic case studies used for expert review as described below.

Electronic case study contents. Programmers created structured case studies from the 1,496 patient records. The data for the case studies came from the KBC Adult Patient Profile (25) where nurses documented a thorough admission assessment from patients and caregivers and the Assessment/Intervention Flowsheet contained documented assessments from each shift throughout the hospital stay (e.g., cognitive status, functional status, fall risk). We selected 71 data elements from these documents based on our prior research and the literature to represent the factors that describe the patients’ health, admission course, and are associated with post-acute needs and outcomes.⁽¹⁰⁾ The case studies provided the experts with a rich description of the hospitalized patient. Missing data was labeled as such in the case study to mimic clinical practice where information is often missing. Study data elements were marked as required in the EHR to minimize missing data.

The Orem Self-Care Deficit Theory provided the organizing framework for the case studies. Orem posits that nursing care is appropriate when the person is not able to engage in self-care.⁽²⁴⁾ This theory appropriately supported our study because we believe that patients who cannot perform self-care after hospital discharge need post-acute care. Orem describes 10 categories of basic conditioning factors that may affect self-care: age; gender; developmental state; health state; sociocultural orientation; health care system factors; family systems factors; patterns of living; environmental factors; and socioeconomic factors. For example, the category of health state included primary diagnoses, co-morbid conditions, medications, procedures, wounds and other elements listed in Table 1.

Table 1.

Case Study Variables

Ability to Learn	Comorbidity Name	# of Discharge Medications
Admission Type	Consciousness Level	Nutrition Risk Eating Poorly
Age At Admission	Discharge Medication	Nutrition Risk Weight Loss
Ambulation Function Changed	Dressing Function Changed	Orientation
Ambulation Current	Dressing Current	Pain Rating Rest
Ambulation Prior	Dressing Prior	Past 6 Month ED Visit
Assistive Equipment Used	Eating Function Changed	Past 6 Month Hospital Stay
Barriers Follow Medication Schedule	Eating Current	Personal History Family History
Bathing Function Changed	Eating Prior	Presence of a Wound
Bathing Prior	Employment Status	Race
Braden Score	Ethnicity	Self-Rated Health
Caregiver Ability	Fall Risk Score	Surgical Procedure Term
Caregiver Availability: day of week	Financial Concern	Toileting Changed
Caregiver Frequency of Availability	Gender	Toileting Current
Caregiver Presence (yes/no)	Hearing	Toileting Prior
Caregiver Availability: time of day	Home Accessibility	Transferring Changed
Caregiver Relationship	Incision Appearance	Transferring Current
Caregiver Understanding	Incision Location	Transferring Prior
Caregiver Willingness	Length of Stay	Vision
Communication Function Changed	Lives With	Wound Type
Communication Current	Living Arrangement
Communication Prior	Marital Status
Caregiver Understanding	Mental Health
Caregiver Willingness	Number of Comorbid Conditions

Multi-Disciplinary Experts. Physicians, and masters or doctorally prepared nurses, social workers, and physical therapists were recruited via snowball sampling to review the case studies and provide their decision making expertise. The experts accessed the case studies via a website built to display their assigned case studies, track their progress, and capture their decisions. A detailed description of expert recruitment, the web-based expert knowledge elicitation process, and Delphi rounds is published elsewhere.⁽²¹⁾

Stratified by geographic region of the United States (north, south, east and west) and discipline (doctor, nurse, physical therapist, social worker), the statistician randomly assigned 171 experts to teams of three to review and judge approximately 13-30 unique case summaries each. Experts were paid as independent consultants per case. Agreement of two out of three experts was accepted as consensus. If all three disagreed on the site of care, the cases went through up to two Delphi rounds seeking majority agreement (2 out of 3). If agreement was not reached the case was not used in the “where to” modeling. Along with their “yes/no refer” decisions and “where to” decisions (i.e. home care, inpatient rehabilitation, skilled nursing facility, nursing home, or hospice), experts identified the factors within each case study that supported their decisions. We used these data to create and validate the two-step expert algorithm.

Data Analysis

We randomly separated the 1496 cases into a training dataset and a validation dataset. We used a number of analytical methods to determine the best model for each decision. These included: 1) Penalized/Regularized Logistic Regression Models performed in R using the glmpath package^{(26, 27)} and 2) classification and regression trees (CART)⁽²⁸⁾ performed in R using regression trees, or the ctree command included in the party package.⁽²⁹⁾ Factors were added to models based on increasing the predictive ability of the algorithm above that already been obtained, while penalizing for the complexity of the model and structure of the data. Separate models were examined where (a) factors were looked at as “blocks” to be included/excluded together, (b) as indicators for level, and (c) based on severity cut-points. The optimal model was statistically determined based on area under the curve (AUC), and Akaike and Bayesian information criterion (AIC & BIC).

Modeling yes or no refer. Patient characteristics selected by experts as important for their decision were not as informative for reducing the data and pinpointing important variables as we had initially hoped. In the final round, for each case by each expert, they selected from one to 71 different items. In all, the experts marked 4,488 characteristics (variables) as important within the 1,496 cases with a mean of 23 per case (SD=10.75). Furthermore, the make-up of important/not-important for referral decision was statistically significant for all variables. Consequently, all 71 variables (Table 1) were initially candidates for inclusion in the prediction models.

Penalized regression uniformly gave the best predictive ability with variable selection based on the BIC criteria. We tested the robustness of the selected variables using 1,000 bootstrap samples. If the predictor did not remain important in at least 50% of the bootstrap samples, its exclusion was tested as a potential refinement of the model. It was retained if its removal affected other variables in the model even if it appeared in <50%. Refinements included: excluding primary diagnosis variables, excluding specific comorbidities, and including combinations of variables as suggested by clustering or CART.

For each of the possible refinements, the ROC curve produced was compared to the ROC curve for the “baseline” model using statistical tests based on both U-statistics and bootstrapping via the pROC package.⁽³⁰⁾ The “best” (least complex) model with similar statistical properties to the “baseline” model was found to be the one with primary diagnosis and comorbidity variables excluded, and included one of the combinations that was suggested by CART.

Modeling Where to Refer. The model for referral location included the variables within 1,204 cases the experts determined the outcome was “yes” refer for care. We excluded 17 cases with uncertain location, even after two Delphi rounds. A number of steps led to the final model: 1) Models for predicting the five referral sites were generated. All methods produced poor performing models, with some models doing worse than chance. 2) Hospice was excluded since only 66 patients were determined as a hospice referral by the experts and the cost of misclassifying a patient to hospice could be high. The main error in prediction was among the facility-based locations. Based on results from steps one and two, we collapsed the referral site outcomes to create the dichotomous referral outcomes of facility versus home care. Since cases who currently reside in a facility were highly likely to return to a facility, these subjects were referred to a facility in the first split of the model. Penalized Logistic Regression models best performed prediction for the remainder of the cases with the outcome facility versus home care.

Results

The 1,496 patients were on average 74.3 years old (11.25 SD, range 55-103). Gender distribution was 55.5% female, 83.4% White and 12.1% Black or African American and 98.3% were non-Hispanic. Among a large variety of primary diagnoses, the most common diagnoses were Pneumonia, Atrial Fibrillation, Acute Exacerbation of Obstructive Chronic Bronchitis, Unspecified Septicemia, and Acute Kidney Failure. Average number of secondary diagnoses and co-existing conditions was 10.29 (range 1 to 32, SD=5.73). Table 2 provides a fuller description of the sample.

Table 2.

Socio-demographic and clinical characteristics of the case study patients. N=1496

Characteristics			Means (SD) [Range] or n (%)
Age			74 (11.25), [55 – 103]
Gender
Male			665 (44.45)
Female			831 (55.55)
Race
White			1247 (83.36)
Black			181 (12.10)
Other			27 (1.80)
Missing			41 (2.74)
Ethnicity
Hispanic/Latino			25 (1.67)
Non-Hispanic Latino			976 (65.24)
Missing			495 (33.09)
Education
Elementary			125 (8.36)
High school			763 (51.00)
College			378 (25.27)
Graduate/Postgraduate			88 (5.88)
Missing			142 (9.49)
Employment Status
Employed			199 (13.30)
Not employed			208 (13.90)
Retired			936 (62.57)
Missing			153 (10.23)
Hospital
A regional			340 (22.73)
B rural			407 (27.21)
C suburban			348 (23.26)
D urban			401 (26.80)
Admit Type
Elective			253 (16.91)
Transfer			20 (1.34)
Emergency			1223 (81.75)
Hospitalized prior 6 months
None			796 (53.21)
Once			346 (23.13)
Two or three times			175 (11.70)
More than three times			55 (3.68)
Missing			124 (8.29)
Number of Medications			8.43 (5.74), [0 – 33]
Co–existing conditions			10.29 (5.73), [1 – 32]
Length of Hospital Stay			6.75 (4.07), [1 – 36]
Self-Rated Health
Excellent			67 (4.48)
Good			584 (39.04)
Average			448 (29.95)
Fair			257 (17.18)
Poor			79 (5.28)
Missing			61 (4.08)
Discharge Disposition
Home to self-care			493 (32.95)
Skilled home care			381 (25.47)
Skilled nursing facility			468 (31.28)
Inpatient rehabilitation			60 (4.01)
Nursing home			45 (3.01)
Hospice			45 (3.01)
Against medical advice			4 (0.27)
Activities of Daily Living
	Ambulation	Bathing	Communicate	Dressing	Eating	Toileting	Transfer
Independent	832 (55.61)	1037 (69.32)	1341 (89.64)	1071 (71.59)	1308 (87.43)	1019 (68.11)	942 (62.97)
Dependent	641 (42.85)	429 (28.68)	122 (8.16)	393 (26.27)	160 (10.70)	446 (29.81)	524 (35.03)
Missing	23 (1.54)	30 (2.01)	33 (2.21)	32 (2.14)	28 (1.87)	31 (2.07)	30 (2.01)

Hypothesis one (H1). H1 was supported. H1-There is a statistically significant correlation between the evidence-based factors (clinical information) and the experts’ yes/no referral decisions. Of the 71 case summary variables, 16 variables (each associated with particular values or interaction terms), were identified as significantly correlated and important to the decision of whether or not to refer the patient for post-acute care. Significant variables include for example employment status, fall risk, several activities of daily living, and number of comorbid conditions. An optimal cut off score indicated the difference between refer and do not refer. Table 3 lists the variables important for the yes/no refer prediction and the percentage of 1,000 bootstrap samples in which they appeared. The remaining factors were discarded as their inclusion did not sufficiently improve the predictive ability of the algorithms while increasing the complexity of the models. In some instances, the same variable with different values yielded significant correlations.

Table 3.

Predictors of the expert’s decisions to refer patients for post-acute care or not and the percentage of 1000 bootstrap samples in which they appeared.*

Factor	% Bootstraps
Employment Status: retired or working	92.5
# of hospital stays within past 6 months	51.2
Fall Risk Score	97.6
Equipment	94.9
Home Accessibility	52.8
Wound Present	68.3
Ambulation Current-level of ambulation function on admission	99.9
Ambulation Change-level of decline in ambulation function from level A to B by discharge	85.7
Ambulation Change-level of decline in ambulation function from level B to C by discharge	58.3
Transfer Change-level of decline in transfer function from level A to B by discharge	80.3
Transfer Change-level of decline in transfer function from level B to C by discharge	81.0
Bathing Change-level of decline in bathing function from level A to B by discharge	84.6
Eating Prior-level of eating function on admission	50.8
Number of Comorbidities	98.0
Caregiver Presence: yes/no	56.1
Discharged on Narcotics	52.9
Interaction Ambulation Current X Transfer Change X Caregiver Presence	99.9

*detailed co-efficients, cut off scores, and levels of measurement are omitted to protect intellectual property

Hypothesis two (H2). H2 was partially supported. H2-There is a statistically significant correlation between the evidence based factors and the site of referral. Models for predicting the five referral sites (home health care, in-patient rehabilitation facility (IPF), skilled nursing facility (SNF), nursing home (NH) and hospice were generated but they proved to be poor performers. Based on the results, we collapsed the site outcomes to depict facility (SNF, IPF, NH) versus home health care referral thus achieving a satisfactory model. For this model there are 13 variables, each associated with particular values or interaction terms, identified as important to the decision of where to refer those patients. Significant variables include for example caregiver availability, pressure ulcer risk, and several activities of daily living, and number of comorbid conditions. An optimal cut off score indicated the difference between home health care referral versus facility. Table 4 lists the variables important for the where to prediction and the percentage of 1,000 bootstrap samples in which they appeared. The remaining factors were discarded as their inclusion did not sufficiently improve the predictive ability of the algorithms while increasing the complexity of the models. In some instances, the same variable with different values yielded significant correlations.

Table 4.

Predictors of the experts’ decisions of where to refer the patients for post-acute care (home care versus facility care) and the percentage of 1000 bootstrap samples in which they appeared.*

Factor	% Bootstraps
Braden Score (pressure ulcer risk)	95.0
Fall Risk Score	60.1
Ambulation Current-level of ambulation function on admission	74.3
Ambulation Current-level of ambulation function on admission	95.6
Ambulation Change-level of decline in ambulation function from level A to B by discharge	85.8
Transfer Current-level of transfer function on admission	37.6
Transfer Change-decline in level of transfer function in level A to B by discharge	81.6
Toileting Current-level of toileting function on admission	64.3
Toileting Current-level of toileting function on admission	60.6
Bathing Current-level of bathing function on admission	71.3
Bathing Change-level of decline in bathing function from level A to B by discharge	69.8
Eating Prior-level of eating function on admission	52.0
/Cableregiver Presence: yes/no	82.0
Caregiver Availability: time of day	87.9
Caregiver Relationship	94.1
Interaction: Ambulation Current X Transfer Current	95.7

*detailed co-efficients, cut off scores, and levels of measurement are omitted to protect intellectual property

Hypothesis three (H3). H3 was supported. H3-The model predicts the expert referral (yes/no) decision with sensitivity, specificity and positive predictive value of greater than 80%. The Area under the Curve (AUC) was 91.5%, sensitivity was 90.1% (95% CI: 88.1-91.9), specificity was 76.9% (95% CI: 71.0-82.0), positive predictive value was 94.2% (95% CI: 92.5-95.6), negative predictive value 65% (95% CI: 59.2-70.6). While specificity was slightly lower than 80%, increasing specificity resulted in less acceptable sensitivity and positive predictive values and a value of >70% was determined to be highly acceptable in this case, since the risk is to over refer which would not do harm to patients, however, it may have cost implications. The training set had 1251 cases; the validation set 245 cases.

Hypothesis four (H4). H4 was partially supported. H4-The model predicted the type of setting (facility vs home care) with sensitivity, specificity and positive predictive value of greater than 80%. The AUC was 89.7%, sensitivity was 89.2% (95% Ci: 84.0-93.2), specificity was 68.0% (95% CI: 53.3-80.5), positive predictive value was 91.6% (95% CI: 86.7-95.1), negative predictive value was 61.8% (95% CI: 47.7-74.6). While specificity was lower than 80%, increasing specificity resulted in less acceptable sensitivity and positive predictive values and a value near 70% was determined to be acceptable in this case; since the risk is to refer to facility care versus home care which would not do harm to patients, however, it may have cost implications. The model was fitted to the validation dataset, using the same beta coefficients and cut-point as found in the validation dataset.

Discussion.

In this study we developed and validated a two-step expert derived algorithm to guide decision making for which patients need a post-acute care referral and which type of setting best matches their needs. Discharge referral decision support algorithms are useful to remove barriers to care imposed by subjectivity and variation in decision-making. We derived our algorithm through expert consensus based on patient need rather than on policy or insurance, which can serve as sources of bias or barriers to appropriate care. The importance of this is shown by Lockery and colleagues who found that patient, family or professional involvement in the DP process did not directly affect discharge placement. Rather DC placement may more accurately reflect insurance status.⁽³¹⁾ For those in lower socioeconomic groups with Medicaid that provides less restrictive home health benefits their care is shifted toward home based services, while Medicare policies support institutional care such as SNF or IRF. We purposely did not provide insurance information to our experts and we instructed them to base their decisions solely on need.

More recently, with the advent of the Affordable Care Act with Bundled Payment Programs and Accountable Care Organizations, insurers and providers are seeking the lowest cost site of care rather than the site that may best meet patients’ needs. Such policy shifts reinforce the need for and value of evidence-based decision support that identifies patients based on their clinical characteristics and not on insurance or local conventions. Over time, comparisons of patient outcomes when our algorithm recommends care versus where patients actually go after discharge will demonstrate the impact of such policy shifts on patient outcomes.

Our algorithm may be helpful to discharge planners in assisting with the assessment of patients for post-acute care. A survey of 37 social workers conducting DP in 36 hospitals reported that assessment was their most important and time consuming task with assessment of home support and help with activities of daily living (ADLs) the most demanding and they spent less time on counseling and more time on concrete tasks such as determining services.⁽³²⁾ Automating the assessment of these two concepts with CDS may be of great value in decreasing the work and cognitive load for discharge planners. Wolock and colleagues found discharge planners dealt with psychosocial problems and relationship issues only when they interfered with discharge and large caseloads prevented discharge planners from having enough time to do it all.⁽⁷⁾ CDS could lift some of this burden by automating the assessment process, alerting discharge planners of high need patients and recommending levels of care as first steps and a “heads up”, thereby allowing more time for the important counseling interventions that engage patients and caregivers in shared decisionmaking. Our algorithm operates off of data collected by nurses during routine patient care at admission and throughout the stay, making efficient use of information and negating the need to spend time collecting it again.

The factors deemed important within the algorithm reflect need for post-acute support and those associated with risk for readmission and other poor discharge outcomes. The algorithm cumulatively brings these factors together to determine the patients’ need for post-acute care and suggest whether a facility level (SNF, IPF, NH) or home health care is most likely to meet their needs. The algorithm heavily weighs the performance of ADLs as critical factors. Multiple studies reinforce the importance of ADL function demonstrating the association between ADL limitations, discharge placement and discharge outcomes.^{(33, 34)}

In our study, experiencing a decline in ambulation, transferring, and bathing was associated with referral to facility-based care versus home care. Similarly, Mason and colleagues found that the need for skilled nursing care or occupational therapy versus custodial care for ADLs differentiated patients discharged with home care versus facility level care respectively.⁽³⁵⁾ Similar to our model, an algorithm that predicts SNF or inpatient rehabilitation versus discharge to home for colorectal cancer surgery patients includes number of comorbidities and number of previous hospital stays as predictors.⁽³⁶⁾ Unique to our model, having a caregiver was important to the referral decision and having a caregiver who was available 24 hours a day/ seven days per week was important regarding the level of care, facility versus home based.

Our previous algorithm, the D2S2 identifies patients who need post-acute care.⁽¹⁰⁾ Translation to practice demonstrated significant declines in readmissions when using the D2S2 compared to not, after controlling for covariates and time.^(37,38) This study built on that work with a larger, more diverse sample and robust set of data elements and created a two-step algorithm to determine the level of care for those identified for referral. The data elements are available shortly after admission and require no additional data collection. We are testing the new algorithm in a quasi-experimental, pre/post design and comparing patient outcomes using propensity modeling in two hospitals.

Study limitations include a sample limited to 1,496 patients that contains only the most common diagnosis for hospital admissions. We also could not account for every patient factor that may be important to these decisions. For example, the patient and caregiver preferences were not included. Also, the experts were unable to “see” the patient but made their decisions based on a written case study and they made their decisions independently whereas in real life they often collaborate as a team.

Conclusion.

Using electronic health record data and expert opinion, our team successfully built and validated a two-step prediction algorithm to support discharge referral decision-making. We achieved highly satisfactory predictive summary statistics on both steps of the algorithm and the factors within it match those commonly identified as associated with post-discharge service need and outcomes. Evidence-based CDS for discharge planning may alleviate some of the workload for discharge planners enabling them to spend more time counseling patients and engaging in shared decision-making. The algorithm was embedded into the EHR of a hospital and tested in a quasi-experimental pre/post design. Next steps include analysis of the experimental data to determine whether the algorithm has an impact on referral rates, site of referral, types of patients referred, and most importantly patient outcomes such as readmission and emergency department use.