Abstract
This paper describes the results of using a modified Delphi approach designed to achieve consensus from eight discharge planning experts regarding the decision to refer hospitalized older adults for post-discharge follow-up. Experts reviewed 150 cases using an online website designed to facilitate their interaction and efforts to reach agreement on the need for a referral for post-discharge care and the appropriate site for such care. In contrast to an average of eight weeks to complete just 50 cases using the traditional mail method, the first online Delphi round for 150 cases were completed in six weeks. Data provided by experts suggest that online Delphi is a time efficient and acceptable methodology for reaching group consensus. Other benefits include instant access to Delphi decision results, live knowledge of the time requirements and progress of each expert, and cost savings in postage, paper, copying, and storage of paper documents. This online Delphi methodology is highly recommended.
INTRODUCTION
Hospital discharge referral decision-making is a complex, multidisciplinary process with many barriers, challenges, and shortfalls. Shortened lengths of hospital stay, inconsistent assessment criteria, and varying levels of expertise and risk tolerance in clinical decision making are among the barriers that impede the ability to make accurate referral decisions 1, 2, 3 Older adults with multiple comorbid conditions and complex treatment regimes add to the challenge. Inadequate discharge planning and referral decision-making can result in elders being discharged before the full effect of treatment is evident, the patient and family fully understand the illness or the treatment plan, and the patient can assume self-care. 2, 3, 4 Some believe the problem is so widespread that virtually all patients older than 60 leave the hospital with unmet needs.3 The care of these older adults is rapidly transferred from formal hospital care to the informal care of unskilled family caregivers. When referrals for follow-up professional care are missed, older adults with unmet needs suffer costly re-hospitalizations, emergency room use, difficulties with activities of daily living, and declines in self-reported health.5, 6, 7, 8, 9 These adverse outcomes are very costly to the older adults, their families, and society, therefore it is critically important to understand and improve the process by which referral decisions are made.
To address these issues, our team is in year two of a National Institute of Nursing Research funded study to develop an expert system to support discharge referral decision-making. The system will assist multidisciplinary clinicians to collect and synthesize key information to support accurate decision making when referring patients for services in home care, inpatient and outpatient settings, and nursing homes. National and local experts in discharge planning are identifying the factors that drive post discharge referrals. We are using these factors to develop a rule-based decision support system. In order to ensure that the knowledge base is as robust as possible and that it represents the array of best practices proposed by the experts, we developed a modified online Delphi technique to reach consensus. Consensus means that all eight experts agree on the referral decision and site of care for each case. The purpose of this paper is to share our online Delphi methodology and to discuss the results and benefits we realized by using this approach to support the goals of the parent study.
METHODS
Expert profiles
The goal of the parent study is to develop a validated knowledge base for discharge planning decision support. In order to meet this goal, we included representatives from all of the clinical domains involved in discharge planning, including nursing, physical therapy, social work, and medicine. These individuals include four nationally known scholars and four local clinical discharge-planning experts. Their charge is to make referral decisions and identify the factors that drive discharge referral decisions for a series of case studies, described below.
Each of the experts was required to have at least five years of clinical or research experience in discharge planning or decision making for older adults. Experts were chosen to ensure diverse viewpoints with national, scholarly, local, and clinical perspectives. The national and scholarly perspectives came from four experts from different areas of the country including the Northeast, Midwest, and West. The four local experts provided the clinical perspective with their knowledge about the day-to-day operations, workflow, and issues about discharge decision-making.
Case studies
We generated a series of 200 case studies from the clinical and research files of adults aged (65 – 90) who were hospitalized for a variety of medical and surgical diagnoses. These diagnoses included congestive heart failure, respiratory tract infection, coronary bypass surgery, cardiac valve replacement, major small and large bowel procedures, angina/myocardial infarction, and orthopedic procedure of the lower extremities. Cases were generated from three previous clinical trials that evaluated transitional care. Data collected on each subject in these studies included medical and surgical history, medication and other therapies, adverse events, self-rated health, functional status, depression, health care utilization, and social support. These data were summarized to create individual case studies, which were presented in a standardized structured format. The case studies were mailed to each expert for discharge assessment. This assessment was limited to four possible discharge dispositions: home care, inpatient facility, outpatient care, or nursing home.
This initial assessment indicated that there was considerable lack of agreement among the experts with regard to discharge disposition. At this point, we decided to employ a Delphi approach to facilitate consensus building.
The Delphi technique
The Delphi methodology is a predominantly qualitative approach in which expert opinion is determined through iterative rounds where respondents individually receive feedback about group opinions. Since information is shared confidentially and without personal contact between respondents, the biasing effects of peer pressure, seniority, or personality are minimized. 10 This technique is typically implemented asynchronously using mail or other media.
For the initial expert evaluations, eight packets containing 50 cases each were copied, collated, and mailed to and from the experts. This was repeated four times at an average cost of $15.00 per package until all 200 cases were judged ($960.00). After all of the cases were mailed back to us, the research assistant entered the referral decisions and the reasons for referral into a database. This entire process took eight months with two additional months for data entry. Knowing that it would probably take at least three Delphi rounds to reach consensus on the 200 decisions, we decided to investigate an alternative approach.
In our study, we used a Delphi technique that facilitated interaction between the experts in an online environment. In our online Delphi approach, the goal is to reach consensus on the 200 cases for the Yes/No referral decision and the site of referral. The decisions of all 8 experts from the original mailed cases were summarized and shown online in a table to the experts who are asked to consider the new decisions and resubmit their decisions assisted by the new input from their peers. The technique provides convenience, anonymity, and removes dominance of one member over another.11 Participants do not have to travel enabling the use of a national or international sample of experts. This was particularly important to us because we wanted a national perspective and case study mailings were very expensive and time consuming.
Online Delphi rounds have been used successfully with e-commerce experts12 and college students 13, 14 resulting in greater convenience, breadth of discussion, and information transfer than in-person group sessions. Almost all studies use at least two rounds and many use several. 15
Features of the online Delphi system
The Delphi website was developed using the following products: Microsoft SQL Server 2000 database, Microsoft IIS (Internet Information Server) 5.0 running on a Microsoft Windows 2000 server, ASP (Active Server Pages) Classic and ASP.NET, Microsoft Visual Studio.NET and Dreamweaver MX. For security we used Microsoft's Public Key Infrastructure that enables encryption of data transmitted over the Internet. The website includes log-in and password protection, step by step instructions, a drop down box to choose the case, a table showing the decisions of all eight experts by case number while maintaining anonymity, a button to access the original script of the case if they would like to review it, drop down boxes to choose whether to refer or not and space to write the reasons for a referral or reasons for changing from the original decision. All data enters a table displaying the decision by expert and the reasons for the decision. As each case is completed, it is removed from the drop down list. A login timer was added to track the time it takes to complete each case. Figure 1 shows a screen shot of a Delphi table for one case.
Figure 1.
Online Delphi system, consensus table.
The database tracked the yes/no referral decision for each case. If the expert changed either the decision or the site for referral, he or she was asked to note the reasons for the change. The online Delphi web site was designed to capture the data from the rounds directly into a database to enable a quick summary of the decisions and their reasons and provide information to the experts for the subsequent rounds. This feature also enables the research team immediate access to the Delphi results. It is used to monitor the expert’s progress and to indicate which expert may need to be reminded to complete the task on time.
Procedure
Prior to opening the website to the experts, the contents were pilot tested with the team of investigators and one expert. The feedback was used to improve the site. For example, important items in the instructions were clarified and constraints added to assure that the experts provided reasons for referral.
The experts were e-mailed instructions on how to access and login to the website. The first 50 cases were placed on the website and the experts were asked to complete them in two weeks. The other batches of 50 cases were added as the prior ones were completed. Cases in which the experts agreed were not placed on the website. The experts were asked to comment on the functionality of the website. The length of time it took for each expert to complete a case and the time from the start of the Delphi round to the finish were tracked. Frequencies were obtained for the number of times a particular expert changed their mind. The number of cases where the experts reached consensus was compared to the number of cases that remain in disagreement for the yes/no referral decision and the site of care.
RESULTS
The experts have completed 150 cases in round one. They commented that the online Delphi website had “clear instructions”, was “easy to use”, and “interesting to be able to see the decisions and reasoning of the other experts.” They liked the ability to work on the cases at a time convenient to their schedules and not having to carry paper packets with them. Two experts said, “it’s nice to have a link that takes you to the case if you want to review something.” It has taken six weeks to complete a Delphi round on 150 cases. This is compared to the cost and time to prepare, copy, collate and mail the packets of just 50 cases to the experts. In the mail method, preparation time alone took one week and two days each were needed for delivery and return. Further, with online Delphi we have instant access to the data versus needing to enter the new judgments into a database from paper. We estimate the time savings for data entry alone to be several months.
The experts began the Delphi process the day the website was live. They spent an average 70 seconds per case (range 41–101 seconds, SD 22). The experts were given two or three weeks to complete a batch of 50 cases. In reality, with each case taking an average of one minute 10 seconds they could have finished all 150 cases in approximately three hours. The six week time frame could be shortened greatly if more demanding deadlines were set for completion of the task.
Delphi round one resulted in full expert agreement on the referral decision and site of care for 48 additional cases (32% of 150 cases). On the 102 cases that remain without full agreement, there is agreement on the decision to refer for 68 (66%) of them. However, although the experts agree to refer, the site of care remains in disagreement. The frequency of disagreement on site of care indicates that the choice between referral to home care versus an inpatient setting is the most common with 45 out of 68 (66%) being this type, followed by home care versus outpatient (N = 14, 21%), a mixture of all four sites (N = 7, 10%), and home care versus nursing home (N = 2, 3%).
DISCUSSION
The Delphi results thus far are very informative. We are convinced that online Delphi is a time efficient and acceptable methodology for reaching group consensus. The experts gave positive feedback, the time and cost savings is significant, and the convenience of instant access to the data is very valuable. The Delphi results provide guidance for our in-person expert sessions because they show areas of disagreement and specific cases where agreement is a problem. The 66% rate of disagreement between home care and inpatient as the site of referral will be a priority for discussion at the next in-person session. In addition, specific cases where mixed referral sites where chosen will also be discussed. Round two is being prepared where the 102 cases remaining without agreement will be presented again to the experts. They will have the ability to see their original decisions, their decisions in round one, and to vote again. Based on the time data from this round, the deadline for completion of the cases will be much shorter. This valuable finding will lead to an increased rate of study progress.
CONCLUSION
This application of an online Delphi methodology had many benefits for the parent study and is highly recommended to others. This informatics solution is helping us to move toward expert consensus conveniently and efficiently.
ACKNOWLEDGEMENTS
This project is funded by the National Institute of Nursing Research, National Institutes of Health, RO1 NR07674-01
Thank you to Amy Wirth, Research Assistant who proficiently developed and maintains the website.
REFERENCES
- 1.Clemens EL, Hayes HE. Assessing and balancing elder risk, safety and autonomy: Decision-making practices of health care professionals. Home Health Care Services Quarterly. 1997;16(3):3–21. doi: 10.1300/J027v16n03_02. [DOI] [PubMed] [Google Scholar]
- 2.Estes CL, Swan JH. Long term care crisis: Elders trapped in the no-care zone. Newbury Park, CA: Sage Publications. 1993.
- 3.Potthoff S J, Kane RL, Franco SJ. Hospital discharge planning for elderly patients: Improving decisions, aligning incentives (Master Contract 500-92-0048). Minnesota: University of Minnesota Institute for Health Services Research. 1995.
- 4.Bowles KH, Naylor MD, Foust JB. Patient characteristics at hospital discharge and a comparison of discharge referral decisions. J Am Geriatri Soc. 2000;50:336–42. doi: 10.1046/j.1532-5415.2002.50067.x. [DOI] [PubMed] [Google Scholar]
- 5.Mamon J, Steinwachs DM, Fahey M, Bone LR, Oktay J, Klein L. Impact of hospital discharge planning on meeting patient needs after returning home. Health Serv Res. 1992 Jun;27(2):155–7. [PMC free article] [PubMed] [Google Scholar]
- 6.Prescott PA, Soeken KL, Briggs M. Identification and referral of hospitalized patients in need of home care. Res Nurs Health. 1995;18:85–95. doi: 10.1002/nur.4770180203. [DOI] [PubMed] [Google Scholar]
- 7.Arenth LM, Mamon J. Determining patient needs after discharge. Nurs Man. 1985;16:20–24. [PubMed] [Google Scholar]
- 8.Naylor MD, Brooten D, Jones R, et al. Comprehensive discharge planning for hospitalized elderly. A randomized clinical trail. An Int Med. 1994;120:999–1006. doi: 10.7326/0003-4819-120-12-199406150-00005. [DOI] [PubMed] [Google Scholar]
- 9.Naylor MD, Brooten D, Campbell R, et al. Comprehensive Discharge Planning and Home Follow-up of Hospitalized Elders. JAMA. 1999;281:613–620. doi: 10.1001/jama.281.7.613. [DOI] [PubMed] [Google Scholar]
- 10.Jairath N, Weinstein J. The Delphi methodology (Part one): A useful administrative approach. Can J Nurs Admin. 1994 Sep–Oct;:29–42. [PubMed] [Google Scholar]
- 11.Delbecq AL, Van de Ven A. Gustafson DH. In. L Delbecq, AH Van de Ven, DH Gustafson. Group techniques for program planning. Scott, Foresman, and Company, Glenview, Il. 1975.
- 12.Addison T. E-commerce project development risks: Evidence from a Delphi survey. Inter J Inform Manage. 2003;23(1):25–40. [Google Scholar]
- 13.Monolescu D, Schifter C. Online focus group: A tool to evaluate online students’ course experience. The Internet and Higher Education. 1999;2(2–3):171–176. [Google Scholar]
- 14.Benunan-Fich R, Hiltz RS, Turoff M. A comparative content analysis of face-to-face vs. Asynchronous group decision-making. Decision Support Systems. 2003;34(4):457–469. [Google Scholar]
- 15.Crisp J, Pelletier D, Duffield C, Adams A, Nagy S. The Delphi Method? Nurs Res. 1997;46(2):116–118. doi: 10.1097/00006199-199703000-00010. [DOI] [PubMed] [Google Scholar]