Successful electronic implementation of discharge referral decision support has a positive impact on 30 and 60-day readmissions

Abstract

In a quasi-experimental study decision support software was installed in three hospitals to study the ability to scale (spread) its use from one hospital on paper to three hospitals as software, and to examine the effect on 30 and 60-day readmissions. The Discharge Decision Support System (D2S2) software analyzes data collected by nurses on admission with a proprietary risk assessment tool, identifies patients in need of post-acute care, and alerts discharge planners. On six intervention units, with a concurrent comparison group of 76 units, we examined the implementation experience and compared readmission outcomes before and after implementation. The software implementation finished one month ahead of schedule, the software performed reliably. High-risk patients admitted in the experimental phase after implementation of D2S2 decision support had significantly fewer 30-day readmissions (a decrease 22.2% to 9.4%) When high and low risk patients were analyzed together, D2S2 achi8eved a 33% relative reduction in 30 day readmissions (13.1% to 8.8%) and sustained a 37% relative reduction at 60 days. The software, available commercially through RightCare Solutions, was adopted by the health system and remains in use after 22 months. The D2S2 risk assessment tool can be installed easily in existing EHR systems. Future research will focus on how the tool influences discharge decision-making and how its accuracy can be improved in specific settings.

Over 35 million adult patient hospital discharges occur each year in the United States (Centers for Disease Control and Prevention, 2013). With each discharge comes a critical decision to discharge the patient home to their self or family care, or to refer them for post-acute services such as skilled home care, skilled nursing facility, or inpatient rehabilitation. Despite the large number of discharge decisions made each year, there are no national, empirically derived decision support tools to assist in making these important decisions. In addition, the vast majority of hospitals in the United States do not estimate the risk of poor post-acute outcomes in a formal way (Bradly et al., 2012), and various attempts at developing clinical tools for risk stratification have performed poorly (Kansagara et al., 2011).

High quality discharge planning is important because appropriate post-acute support enhances patients’ safety and reduces the likelihood of an avoidable hospital readmission (Bauer, Fitzgerald, Haesler, & Manfrin, 2009; Boutwell, Griffin, Hwu, & Shannon, 2009; Shepperd et al., 2010). The Agency for Healthcare Research and Quality (AHRQ) estimated that Americans spend approximately $30 billion annually on potentially preventable hospital readmissions (Jiang, Russo, & Barrett, 2011). Penalties, legislated through the Affordable Care Act and the Hospital Readmissions Reduction Program, motivate hospitals to conduct interventions that reduce the likelihood of readmission, especially within 30 days (Centers for Medicare and Medicaid Services, 2013).

In a previous pilot study, our research team administered the Discharge Decision Support System (D2S2), and demonstrated that the percentage of high-risk patients readmitted by 30 and 60 days decreased by 6% and 9%, respectively, representing a 26% relative reduction in readmissions for both periods among medical patients on four units of one hospital (Bowles, Hanlon, Holland, Potashnik, & Topaz, 2014). In that study the tool was administered manually on paper.

The purpose of this study was to to develop, install and use the D2S2 electronically as software and to further test the impact of the resulting decision support software on readmission when scaled across three hospitals within a major academic health system. The study was supported by a Small Business Innovation Research (SBIR) grant awarded to RightCare Solutions, a software company that exclusively licenced the D2S2 from the University of Pennsylvania. Aim one was to assess the ability to scale the use of the D2S2 from paper to a software platform that collects, calculates, and delivers discharge referral decision support. Aim two was to evaluate the rate of tool completion by clinicians, and Aim three was to evaluate the impact of the D2S2 on 30 and 60-day all-cause hospital readmissions and time to readmission.

Method

Design

This study was a quasi-experimental, two-phase study (control and experimental) with concurrent comparison units across three hospitals in a large, urban, academic health system. A prospective, randomized, clinical trial at the patient level was not possible because discharge planners are assigned to units and care for multiple patients on the same unit. This would have resulted in contamination and therefore negating the ability to maintain a control condition. In this study discharge planners were registered nurse case managers (CMs) and social workers (SWs).

The information technology intervention studied was the D2S2, a risk assessment tool that provides discharge referral decision support (Bowles et al., 2008; Bowles et al., 2009). An information technology company, RightCare Solutions installed the D2S2 within the health system’s electronic health record (EHR) and the tool was completed in the three health system hospitals by nurses with adult patients upon admission. To limit the scope of the SBIR research project for feasibility, the health system leadership (chief nursing officers and the chief medical officers) selected six medical units across the three health system hospitals to participate in the study. The six units included three units at the largest facility, two units at the second facility, and one unit at the third facility. The remaining hospital units did not receive the decision support intervention so they served as concurrent non-intervention comparison units.

To assess threats to validity of the two-phase design, the study team confirmed with the hospital leaders that no new readmission reduction programs were initiated or halted at this time and that this was an ideal time to test the impact of D2S2. Although many different types of programs were ongoing across the hospital (eg. telephone follow-up, alerts regarding previous admissions, unit-based pharmacists, transitional care model, team huddles on high risk or readmitted patients), the programs were already in place during the control phase and continued during the experimental phase as well. In addition, the study included a concurrent nonintervention comparison group to assess readmission rates over the entire study period on nonintervention units.

Prior to starting the study, the team obtained consent and Health Insurance Portability and Accountability Act (HIPAA) waivers from the Institutional Review Board of the University of Pennsylvania. Waivers were justified as the study was low risk and the research team dealt with de-identified data. The study team did not disclose any personal health identifiers. In addition, RightCare Solutions has a Business Associate Agreement with the health system that maintains stringent HIPAA procedures that assure confidentiality and security.

Sample

Adult patients admitted on and discharged alive from the six intervention units were eligible for the intervention. Patients who were transferred onto the intervention units were excluded because they had not received the intervention on admission.

The six intervention units contained patients with predominately medical primary diagnoses such as: atrial fibrillation, sub-endocardial infarction initial episode of care, coronary atherosclerosis of native coronary artery, acute on chronic systolic heart failure, and paroxysmal supraventricular tachycardia. Patients admitted to all remaining hospital units across the health system (76 units) comprise the concurrent non-intervention comparison group. The predominant diagnoses of patients on the concurrent units were: acute on chronic heart failure, coronary atherosclerosis native vessel, sub-endocardial infarction, diverticulitis, morbid obesity, septicemia, and post-operative infection.

Measures

The D2S2

The Discharge Decision Support System (D2S2) is a six-item risk assessment tool that identifies patients at risk for poor discharge outcomes and recommends referral for post-acute services such as home care, skilled nursing facilities, rehabilitation, or long-term care. The scores can range from 0–11. Because the information is provided shortly after admission, the D2S2 risk score is expected to help discharge planners and other hospital staff with identifying high risk patients, prioritizing and managing the time available for care planning because this information is provided shortly after admission. Having more time to educate high risk patients or their caregivers about the need for post-acute care services, plan for the appropriate level of care, and coordinate a care transition plan is important to be able to reduce 30-day and 60-day readmissions.

Developed in 2009 (Bowles et al., 2008; Bowles et al., 2009), the tool was amended from its original form to improve its ease of use in clinical practice. Validity testing with a hold-out sample demonstrated a good area under the curve (AUC) of .86 (Bowles et al., 2009). The level of the AUC reflects accuracy, the higher the better, with 1.0 being perfect. Ease of use was improved by demonstrating comparable predictive value using a two-item depression-screening questionnaire versus a 20-item tool, allowing us to insert the more feasible, shorter questions.

Two versions of the tool (cognitively intact or impaired versions), each contain six-items. Both tools assess the following patient characteristics: level of mobility, length of stay, overall health-status ratings, and number of comorbidities. Access to caregivers and income level are unique for those cognitively impaired and depression and age are unique in the cognitively intact version.

Discharge Outcomes

Referral to traditional post-acute care (home care, inpatient rehabilitation, nursing home, hospice, or skilled nursing facility) and patient refusal was collected by research assistants through chart reviews of the discharge planners’ documentation and from the hospital database. Readmissions were collected from the admission/discharge/transgfer (ADT) data of all subsequent episodes of care matching on patient name, date of birth, and medical record number.

For this study, a hospital readmission was defined as occurring when a patient is discharged from the applicable hospital to a non-acute setting and then is readmitted within a 30-day or 60-day time period to any of the three hospitals that are part of the large, urban, academic health system. Readmissions occurring outside the health system were not able to be captured.

Software Implementation

In preparation for implementing the software, the RightCare Solutions team assessed the technical and functional requirements of the hospital information system and the nurses’ and discharge planners’ workflows. They concluded that only three points of systems integration were needed. The first requirement was to establish a two-way virtual private network (VPN). This link facilitates secure data transfers between the RightCare software-as-a-service platform and the hospital systems and is crucial for maintaining security since the D2S2 information flowed back and forth between the hospital and RightCare. The second step of the process was to establish real-time flow of Admission, Discharge, Transfer (ADT) messages from the hospital to RightCare so that the RightCare active patient list will be consistently current with the hospitals’ electronic health record (EHR) and other point-of-care systems. ADT messages typically contain details such as dates and location of inpatient episodes and emergency department use.

The final piece of integration was embedding the D2S2 assessment tool into the existing EHR and discharge planning workflow. At the request of the hospital leaders, the D2S2 tool was embedded into the existing nursing admission assessment in the hospitals’ EHR. This fit well into the nurses’ workflow as they completed other assessment questions on all admitted patients. RightCare was able to obtain data for several of the D2S2 questions from other sources so the nurses only had to complete four new items. For example, age and length of stay came for the admission database. To ensure the nurses were accurately entering data into the system and that each active patient was accurately identified in the RightCare software through ADT messages, each month RightCare personnel compared a data extract from the EHR and matched it against the patient encounter and assessment information populated in the RightCare database. Finally, RightCare was successfully able to create a sign-on process that allowed the discharge planners to log into D2S2 with their existing corporate credentials. This minimized the need for another login and password.

In the experimental phase, RightCare processed the D2S2 scores and implemented a daily email alert that informed discharge planners about the high-risk patients on their respective units. To verify the correct scores were provided to the users, a software program generated a test dataset that contained all possible combinations of answer options. This dataset was used to test that the values attached to the tool’s questions were generating the correct D2S2 score. Additionally during testing, a second software program sent emails to development team members who had patients assigned to them to test the messaging feature. In all, 23 test cases confirmed accuracy in the email alerts for content, formatting, and timing. This concluded the methods for aim one.

Data Collection

Control Phase

The control phase, representing usual care, was studied for three weeks to establish a baseline about risk status and readmissions occurring among patients admitted and discharged alive from the intervention and concurrent units. The D2S2 assessment tool was embedded into the existing EHR nursing admission assessment and was completed on 82 admitting units across the health system (76 non-intervention and 6 intervention units). Admitting bedside nurses were educated about the new questions through a voice-over PowerPoint in keeping with their usual mode of receiving educational information. Each individual D2S2 question was marked as “significant” in the health system EHRs using an exclamation mark (!) after each item. This practice is consistent with other required data elements on the nursing admission assessment at the health system, in order to ensure high compliance and completion of the D2S2 assessment questions.

Usual care included an assessment using the D2S2 by the admitting nurses on each floor, but the D2S2 recommendations, scores, and answers were not shared with the bedside nurses or discharge planners. Discharge planners conducted their usual processes, which included interviewing every patient and/or their caregivers, reviewing the medical record to determine the discharge plan, and discussing the plan in daily discharge rounds with hospitalists and bedside registered nurses. They had no interview guide or standardized assessment form for this purpose. The discharge planners and/or hospitalists made the referral suggestions.

Experimental Phase

After the completion of the control phase, the research team trained the intervention units’ case manager and social work staff in-person about the D2S2 questions, scores, factors, and the recommendations. The admitting nurses continued to complete the D2S2 items, but now for the experimental phase, RightCare Solutions sent daily email alerts at 8am Eastern Time to all case manager and social work discharge planning staff assigned to the six intervention units across the three separate health system hospitals.

The email alerts provided a summary number of total patients admitted on their unit identified as high-risk by D2S2 to help them prioritize their work. The message also suggested that these patients receive a referral for post-acute services or some other intervention to mediate their risk. The email contained a hyperlink to a secure RightCare Solutions portal that provided an active patient list and details of the D2S2 risk scores for their respective patients. In this secure portal, each discharge planner was able to see their active patients sorted by risk-score. They viewed the scores, the answers to the screening questions (mobility status, depression, caregiver status etc), and considered the D2S2 advice as they made the discharge plans and referral decisions. The alerts provided information on how patients answered the questions, and recommended a post-acute referral or not based on the D2S2 score. The experimental phase lasted four weeks.

Data Analysis

Due to conflict of interest for authors KHB, MB, MT, and EH, independent statisticians (AH and JC) completed the data analysis and interpretation for the study. To evaluate implementation and success in expanding the use of the D2S2 across a three hospital health care system (aim one), we verified the function of the software, tracked time to implementation, customer calls, length of time the software performed, and response time of the software. To evaluate the rate of tool completion (aim two), the total number of completed assessments was divided by the number of patients admitted, then multiplied by 100. The pre-determined goal was completion of at least 80% of admissions.

To evaluate the impact of the D2S2 on 30 and 60-day readmissions (aim three) and time to readmission, the statisticians (JC and AH) first compared the high and low risk patients on the intervention and concurrent non-intervention units between the control and experimental phases on age, race, gender, length of stay (LOS), D2S2 score, and primary diagnosis to examine for differences. The study team was limited in the number and type of patient characteristics available for baseline comparisons because the health system’s information technology department was unable to provide additional data from the patient record.

The descriptive analyses included means, medians, standard deviations, and histograms for continuous variables and contingency tables with proportions for categorical data. Comparisons were based on t-tests or Wilcoxon rank sum tests for continuous variables, depending on the symmetry of the distributions, and on contingency tables and logistic regression for binary or ordinal variables.

Readmission was evaluated at each of the two time study phases (30 and 60 days after the date of discharge from the index hospital stay) using logistic regression models of outcome regressed on group. The samples were compared within D2S2 score high-risk versus low-risk groups and combined high and low risk D2S2 results. Those recommended for post-acute referral by the D2S2 are high-risk, and those not recommended for referral are low-risk.

Time to readmission since index discharge was analyzed using Cox proportional hazards modeling of time (Cox, 1972). Furthermore, the Kaplan-Meier product limit curve was used to estimate the 30 and 60 day readmission rates between each of the two study phases for the intervention units. The samples were compared within D2S2 score high-risk versus low-risk groups and combined high and low risk D2S2 results. Given that the intervention was “decision support”, the discharge planners were free to override the D2S2 advice or it may have influenced their decisions to refer, so we examined referral rates and site of referral between the two study phases using t-tests and chi square respectively.

Results

Health system wide, in the control phase, nurses completed 3,005 D2S2 assessments for newly admitted patients. In the experimental phase, nurses completed 4,507 D2S2 patient assessments for newly readmitted patients across the health system. Patients admitted on the oncology service and patients expired during hospitalization or transferred from or to a nonintervention unit were excluded from the analysis in both phases leaving a total of 2,816 in the control phase and 3,450 in the experimental phase. Specifically, the samples from the six intervention units in the control phase included 176 patients and the experimental phase included 228 patients admitted to those units and discharge alive. The remaining patients served as the concurrent non-intervention comparison group with 2,640 in the control phase and 3,222 in the experimental phase.

For all patients on the intervention units, the average age was 61 (SD 15.6), the average length of stay was 4.5 days (SD 4.43), and the average D2S2 score was 2.5 (range 0–9.5, SD1.64). On the non-intervention units the average age was 57 (SD17.43), the average length of stay was 4.6 days (SD4.31), and the average D2S2 score was 2.5 (range 0–8, SD1.60). Table 1 shows the comparisons between study phases for high and low risk patients from the intervention and concurrent non-intervention units, including age, race, gender, length of stay, and D2S2 score.

Table 1.

High risk versus low risk comparisons by study phase and units.

	Intervention Unit Patients
	High Risk		Low Risk
	Control Phase (n=54)	Experimental Phase (n=64)	Control Phase (n=122)	Experimental Phase (n=164)
Age (M, SD)	66.13 (16.52)	64.13 (15.75)	59.66 (15.89)	60.09 (14.77)
Race (%)
White	57.41	60.94	71.31	68.90
Black	25.93	29.69	17.21	18.29
Gender (%)
female	51.85	40.63	39.34	33.54
LOS (M, SD)	8.09 (5.49)	6.80 (5.69)	3.61 (3.69)	3.12 (2.70)
D2S2 score (M, SD)	4.91 (1.20)a	4.40 (0.90)a	1.61 (0.86)	1.60 (0.78)
	Non-intervention Unit Patients
	High Risk		Low Risk
	Control Phase (n=818)	Experimental Phase (n=923)	Control Phase (n=1822)	Experimental Phase (n=2299)
Age (M, SD)	64.78 (16.81)	65.72 (17.67)	53.27 (16.33)	53.85 (16.26)
Race (%)
White	46.09	48.00	55.71	55.72
Black	46.09	43.34	34.03	35.54
Gender (%)
female	54.89	53.20	51.65	51.33
LOS (M, SD)	7.15 (6.48)a	5.21 (4.06)a	4.20 (3.91)b	3.74 (3.24)b
D2S2 score (M, SD)	4.57 (1.12)	4.45 (0.97)	1.63 (0.84)	1.63 (0.84)

^ap-value <0.001

^bp-value <0.05

Length of Stay (LOS)

The most common diagnoses for intervetnion unit patients in both study phases were cardiac related (eg. percutaneous cardiovascular procedures without MI, cardiac arrhythmia, heart failure). Although primarily medical patients, 51% of the control phase and 43% of the experimental phase for patients on the intervention units had surgical procedure codes associated with their hospital stay representing procedures such as excision or destruction of other lesion or tissue of heart, endovascular approach; percutaneous transluminal coronary angioplasty; right heart cardiac catheterization; right and left heart cardiac catheterization; and central venous catheter placement with guidance.

Aim one-Decision Support Implementation

Aim one was to scale the use of the D2S2 decision support from a paper-based tool to a software platform that collects, calculates, and delivers discharge referral decision support. Implementation was proposed to take 12 weeks and was achieved within eight weeks and the software continues to operate in the health system hospitals 22 months later. From the time the software went live it performed reliably by being available 100% of the time, indicating no software failures. D2S2 web-portal response time was rapid at 0.2 sec average (0.6 sec max) indicting the clinicians could rapidly access the screens with minimal wait time. There were only two reported and validated bugs in D2S2, only five customer support calls to the RightCare help line, and only 16 support request emails to RightCare over the entire study period indicating people had few problems or questions when using the software. Aim one was fully achieved.

Aim 2-Clinician Tool Completion Rate

Aim 2 was to evaluate the rate of tool completion by clinicians to determine whether the admitting nurses were completing the tool as instructed. The threshold of success regarding the tool completion rate by clinicians was set at 80%, but we achieved 87% in the control phase and 90% in the experimental phase. This means the nurses completed the D2S2 tool on 87–90% of the admissions to the hospital on the intervention units. The reason for incomplete assessments was most often a patient was unable to answer with no family member present.

Aim 3-Readmission Rates and Time to Readmission

Readmission Rates-Intervention Units

There was a statistically significant improvement in 30-day readmissions among patients on the six intervention units where the D2S2 identified high-risk patients and the information was provided to the discharge planners compared to those in the control phase (Figure 1). The greatest impact was achieved with the high-risk patients. The 30-day readmission rate for intervention unit high-risk patients in the control phase was 22.2% and there were fewer readmissions (9.4%) during the experimental phase (a 58% relative reduction; p<0.05). Overall (combining high and low risk patients), the experimental phase using the D2S2 achieved a 33% relative reduction in 30 day readmissions compared to patients in the control phase (13.1% to 8.8%, p=0.015). In the control and experimental phases 9% and 8.5% of patients scoring low risk on the D2S2 were readmitted respectively.

Figure 1.

30-day readmissions on the intervention units control versus experimental phase

The analysis for 60-day readmissions shows a continued impact on readmissions. In the high-risk patients, 31.5% were readmitted by 60 days in the control phase compared to 15.6% in the experimental phase (a 50% relative improvement). Overall (combining high and low risk patients) the experimental phase resulted in a 37% relative improvement in readmissions at 60 days (Figure 2).

Figure 2.

60-day readmissions on the intervention units, control versus experimental phase

Readmission Rates-Non-Intervention Units

To increase the validity of the quasi-experimental study design, the readmission rates were compared between patients admitted in the control phase (n=2,640) and the experimental phase (n=3,222) on hospital units not designated for the decision support intervention (concurrent non-intervention comparison group). The readmission rates on the non-intervention units for high-risk patients increased 0.4% over the same period as the study phases (Figure 3). Overall, for high and low risk patients combined, the readmission rate improved only 0.5% percent (13.1 to 12.6%). This shows that no other interventions affected the readmission rates since they did not decrease or increase substantially over the study periods.

Figure 3.

30-day readmissions on the non-intervention units, control versus experimental phase

Time to Readmission-Intervention Units

The survival analysis (Cox Model) compared high and low risk patients within control and experimental phases on the intervention units. In the control phase we see higher proportions of high risk patients readmitted over time than low risk patients (Figure 4). But, during the experimental phase, the proportion of high and low risk patients readmitted are almost identical (p=.928, Figure 5). The proportion of high risk patients readmitted over time decreased to the level seen among low risk patients.

Figure 4.

Time to readmission for high-risk versus low-risk patients, control phase on the intervention units

Figure 5.

Time to readmission for high-risk versus low-risk patients, experimental phase on the intervention units

When comparing the survival curves of high-risk patients in the control phase to high-risk patients in the experimental phase, there is a significant improvement in time to readmission. The top line in Figure 6 shows more high-risk patients in the control phase readmitted sooner than in the experimental phase. In the experimental phase when the D2S2 was used to notify discharge planners about the high-risk patients, the numbers of readmissions among high risk patients over time is significantly lower (lower line) (p=.037).

Figure 6.

Time to readmission for high-risk patients, control versus experimental phase, on intervention units

Time to Readmission-Non-Intervention Units

On the non-intervention units, where the D2S2 was collected but not shared, we know which patients scored high versus low risk on the D2S2. In Figures 7 and 8 below, one can see that the D2S2 differentiates between high (top line) and low risk patients (bottom line) on time to readmission with no change over time due to no decision support intervention (P<0.0001 in both phases). This shows the ability of the tool to differentiate those at risk upon admission. In addition, comparison of high risk patients between control and experimental phases shows the relative stability of readmissions over the time of the study throughout the rest of the hospital, serving as a concurrent non-intervention comparison group (p=.688), Figure 9).

Figure 7.

Time to readmission for high-risk versus low-risk patients, control phase on the non-intervention units

Figure 8.

Time to readmission for high-risk versus low- risk patients, experimental phase on the non-intervention units

Figure 9.

Time to readmission for high risk patients, control versus experimental phase on the non-intervention units

Referral rates and site of referral

In the control phase, 53% were offered post-acute care while 47% were offered referrals in the experimental phase (p=.226). The case managers referred high-risk patients identified by the D2S2 to traditional post-acute care sites 78% of the time in the control phase when the D2S2 recommendations, scores, and factors were blinded to case managers and 67% of the time in the experimental phase (p=.146) when the D2S2 recommendations, scores, and factors were shared with case managers.

The site of post-acute care referral for all patients high-risk and low-risk varied between the control and experimental periods but was not statistically significant (p=.096). In the control period, 47% were discharged home to self-care; 26% went home with skilled home care; 7% to inpatient rehabilitation; 13% refused post-acute care. In the experimental phase 55% were discharged to self-care; 19% went to home care; 4% to inpatient rehabilitation; 18% refused post-acute care with the remainder distributed between hospice, home infusion and transfer to another acute care facility in both periods. Referrals of high risk patients to other services such as psychiatric services, outpatient physical therapy, transitional care, nurse navigator, or follow-up phone calls may have occurred but were unable to be tracked for this study.

Discussion

The findings of this study provide evidence of the ability to scale the use of the D2S2 decision support tool via software and to demonstrate the effectiveness of the D2S2 to decrease 30 and 60-day readmissions. The first aim was to scale the use of the D2S2 from paper to a software platform that collects, calculates, and delivers discharge referral decision support. The programmers achieved this aim four weeks ahead of the proposed schedule and the software performed reliably. The team achieved integration into the workflow of the discharge planners, which is a critical requirement for decision support success (Horsky et al., 2012; Yuan, Finley, Long, Mills, & Johnson, 2013).

Aim two was to evaluate the rate of tool completion by clinicians. The study exceeded the pre-set threshold of 80% with between 87–90% of admissions having a completed the D2S2 assessment within 24 hours of admission. This success is explained largely because the D2S2 data elements are quick and easy to answer, the number of questions is few, and they were marked with an exclamation mark (!) which means they are a mandatory element in the nursing assessment flow sheet. Since several of the questions were pre-populated from other places in the EHR leaving only four new items to be documented by the admitting nurses, the workload was light. In addition, the study took place at an academic health system where clinicians are often asked to collect data for research.

Together the achievement of aims one and two demonstrates the ability to successfully scale the D2S2 software across multiple facilities, while maintaining a high data capture rate by clinicians and reliable software performance. For aim three, the D2S2 had a statistically significant and clinically meaningful impact on patients admitted to units that used D2S2 with a 33% and 37% relative reduction in 30 and 60-day readmissions respectively for high and low risk patients combined. The largest effect was seen with high risk patients where a 58% relative reduction was achieved.

Evidence-based tools like D2S2 are much needed because nationally there is great variability in risk tolerance and decision making regarding referral decisions from acute care; some places may over refer wasting precious resources, while others under refer leaving patients in need without services. Several readmission risk prediction models have been developed and tested. Despite these efforts, most readmission risk prediction models have poor discriminative ability (c-statistic 0.55–.72) and minimal progress has been made. (Kansagara et al., 2011) Risk prediction models have traditionally focused on administrative data and resource utilization collected after the hospital discharge. A large limitation of these models is they occur too late in the care process and they lack the social determinants associated with health including income, insurance status, caregiver availability, access to care, discharge location and patient functional status. The D2S2 includes most of these important factors and is collected upon admission.

The real purpose of the D2S2 is to get patients the care they need to support them after discharge, rather than to predict readmission. The D2S2 alerted the discharge planners about patients in need of post-acute care within 24 hours of admission. This may have provided more time to focus on the highest risk patients and to engage the family and care team in planning. In addition to the alert, the discharge planners received a list of patients sorted by risk score and saw the answers to the D2S2 questions for each patient. The discharge planners were encouraged to use this list to prioritize their workday, visiting the highest risk patients first. Having the answers to the D2S2 questions about important variables such as depression and self-rated health may have helped the discharge planners tailor the plans more effectively. Depression and self-rated health are consistently associated with risk for poor outcomes.(Biering, Bøtker, Niemann, & Hjollund, 2014; Cancino, Culpepper, Sadikova, Martin, Jack, & Mitchell, 2014; Coleman, Min, Chomiak, & Kramer, 2004; Ketterer, Draus, McCord, Mossallam, & Hudson, 2014; Sharif, Parekh, Pierson, Kuo, & Sharma, 2014) The consistency in this data and the 2011 study where there was a 26% relative reduction in both 30 and 60-day readmissions is encouraging and gives us confidence in the reproducible effect of the D2S2 (Bowles et al., 2014).

Prior to this study, the discharge planners did not use a standardized method for making post-acute care referral decisions. This could lead to great variation in decision-making quality due to variations in risk tolerance, assessments, clinician skill, and availability of clinical information (Bowles, Naylor, & Foust, 2002; Bowles, Foust, & Naylor, 2003) The D2S2 may have decreased the variability and cognitive load on the discharge planners and helped them to better identify, prioritize, and plan for post-acute support. Future work should include a qualitative component with the discharge planners to understand how their decision-making may have changed with the D2S2 advice.

The D2S2 provides decision support versus a mandate for post-acute care. Therefore, as expected, discharge planners did not refer every patient identified by the D2S2 for post-acute care, and they did refer some patients where the D2S2 did not indicate the need. However, as shown in figure 5, the tool clearly differentiated high from low risk patients during the usual care phase. Once we started making the clinicians aware of the high risk patients the readmission rates associated with those patients dropped. Clinicians may have referred high risk patients to other transitional care programs than the traditional post-acute settings accounted for in the hospital database for this study. They may have done more teaching, earlier referrals, or consults to psychiatric services or physical therapy. In addition, we do not know whether knowing the D2S2 scores at admission and the details of the factors changes the referral decisions by the time discharge arrives. For example, patients scoring as having a mobility deficit might be fitted with a walker or cane while an inpatient, or the positive score for depression might result in a referral for inpatient or outpatient behavioral health consults making the need for formal referral to home care no longer necessary. We also recognize that the clinicians may consider additional risk factors not captured by the D2S2.

Future work should also distribute the decision support to all members of the interprofessional team because discharge referral for post-acute care is an interdisciplinary decision. In addition, further work is needed to improve the sensitivity of the D2S2 because 8.5%–9% of those labeled low risk were readmitted and therefore may actually have been high risk. Ongoing analysis of the characteristics of low risk patients who are readmitted will guide adjustments to the assessment tool’s scoring or the addition of other factors to improve the performance of the decision support tool.

Limitations

The study is limited to three hospitals within one academic health system and the data on readmissions did not include those occurring outside of the health system. Some of the patients may have been readmitted to a hospital outside the health system making the patient readmission rate higher than this study’s reported institutional rates. In addition, the intervention units contained mainly medical patients so future studies should either focus on or include a larger surgical population.

Similar to other information technology interventions, we were unable to conduct a randomized controlled trial, but we did include a concurrent control group to strengthen the interpretation of study findings. Also, we were limited in the type of data we had to describe and compare our samples; therefore there may be some confounding characteristics that we were unable to control for such as the quality of their caregivers, patients’ self-care skill, or the influence on the discharge planners of knowing patients were high risk.

Beyond the traditional discharge destination sites of home care, inpatient rehabilitation, skilled nursing facility, nursing home, hospice, and home to self-care, we are unable to report the exact interventions that high-risk patients received therefore a report of how well the D2S2 advice matched clinician action is difficult to calculate.

More research is needed to understand how decision-making is impacted and how discharge planners use the information when making plans with patients. Future use of the D2S2 includes expansion to additional hospitals with enhancements that improve its accuracy in each specific setting, and support connectivity to post-acute care settings. This work is underway to meet the aims of a Phase 2 SBIR granted funded by the National Institute of Nursing Research (R44 NRO13609). The software is available commercially through RightCare Solutions.

Conclusion

While other have focused on predicting readmission, our approach is to enhance identification of patients in need of post-acute care while others have focused on predicting readmission. The D2S2 helps clinicians target patients at risk for poor discharge outcomes so we can match services and meet their needs to keep them in the community longer. Installation of the D2S2 into existing EHRs can take two months or less; nurses and discharge planners adopted its use into their workflow, and the software functions reliably. Thirty and 60-day readmissions improved significantly during the period of decision support use in a three hospital academic health system.