Abstract
Introduction: Electronic referral (eReferral) systems have been designed with the dual purpose of decreasing wait times and improving workflow efficiency. Evidence about the clinical and economic value enabled through the use of eReferral systems is limited. Our objective was to review the evidence base for the effectiveness and cost-effectiveness of eReferral systems. This review is part of a bigger project to inform the economic benefits of a regional eReferral implementation program.
Material and methods: A systematic search was conducted to capture the available literature on the effectiveness and cost-effectiveness of eReferral system interventions. Evaluation of eReferral system for cost or outcome(s) were included. Strictly e-consultation systems were excluded. We only included publications in English.
Results: We found 274 citations. After removing duplicates and conducting levels one and two screenings, nine publications qualified. Results were divided into four categories: cost or cost-effectiveness analysis, changes in workflow efficiency, the quantity of referrals, and the quality of referrals. A full economic evaluation, conducted in Denmark, found that an eReferral system was cost-effective compared with a paper-based referral system. Of the other eight studies, three demonstrated positive changes in referral processing; two evaluated changes in the quality of the referrals, and three evaluated if the eReferral system increased the quantity of referrals.
Discussion: The evidence base on the effectiveness of eReferral systems to improve communication between primary care and specialists and to decrease wait times is positive but limited. Economic evaluations are needed to examine the clinical and economic value of eReferral systems in health care.
Keywords: electronic referrals, eHealth systems, review, costs, cost-effectiveness
Introduction
Timely access to specialist care is critical to achieve better health outcomes and reduce health care costs for patients and the health system.1 Access to specialists depends on several factors, including a patient’s clinical condition, a patient’s age, the type and location of the referring clinic, a specialist’s availability, as well as the level of communication and type of information shared between primary care physicians and specialists in the referral process.2,3 Paper-based referrals by fax, the standard process in many practices, have been associated with referral and booking delays due to incomplete or missing information such as patient’s data, clinical laboratories, X-ray images.4
Electronic referral (eReferral) systems have been designed to improve wait times and efficiency by electronically standardizing information and communication within the referral process.5 According to Liddy (2015), electronic referrals allude to the automation of the referral process in which appointment and other information regarding the consultation are transferred between two or more health care providers.6 The results of a recent systematic review by Naserias et al (2015) of related e-consultation systems indicated considerable variability in success factors such as improving access to specialist care, reducing wait times, and improving quality of the referral process.7 The review identified 27 publications reporting on a mix of e-consultation program solutions and related eReferral systems (Norway, England, Denmark, The Netherlands, New Zealand). Most of these programs were implemented in single site settings (eg, hospitals) and have not been implemented at regional or national levels. Given the additional functions of e-consultation solutions, their evidence cannot be translated to the more common-place eReferral systems that do not support integrated telemedicine consultation. Compared to the literature on e-consultations (see, for example, review by Heimly et al, 2009), no review has focussed on the evaluation of eReferral systems from a cost and benefits point of view and it is currently unknown if eReferral systems are cost-effective strategies.8
Our objective was to describe and appraise the evidence-base on the effectiveness and cost-effectiveness of eReferral systems. This review is part of a bigger project to inform the economic benefits of a regional eReferral implementation program. A secondary goal of this review was to inform future evaluations of eReferral systems. Based on expert consensus, we hypothesized that the evidence-base for eReferral system interventions was limited and inconclusive.
Material and methods
Search strategy and inclusion/exclusion criteria
A search was conducted to identify literature reporting on the incremental cost and cost-effectiveness as well as changes in quality, quantity, and efficiency of referrals associated with eReferral systems from PCP to specialists compared to paper-based referrals in the general population. The search was conducted using PubMed and OVID databases (Embase since 1974 and OVID Medline Epub Ahead of Print, In-Process & Other Non-Indexed Citations, Ovid MEDLINE(R) Daily and Ovid MEDLINE(R) since 1946). Different combinations of MeSH terms and keywords were used to find the most relevant studies without date of publication restriction (see Table 1). Only publications in English were included. The search was conducted from October to December 2016 and updated in June 2018. In addition, a gray literature search was conducted in parallel to identify reports, books, and websites regarding evaluation of electronic referral systems compared with paper-based process. The gray literature review was performed using the Google search engine based on similar keywords and terms used in the PubMed and OVID database searches (Table 1).
Table 1.
Search | Key term | Term 2 | Complementary term | ||
---|---|---|---|---|---|
#1 | Electronic referral | AND | - | OR | Referral |
#2 | Electronic referral | Cost | Referral/cost analysis | ||
#3 | Cost-benefit analysis | Electronic referral | Cost-effectiveness/referral | ||
#4 | e-referral | - | - |
Selection process and data analysis
Studies retrieved by the search strategy were first screened for inclusion and exclusion criteria based on a review of the title and abstract by one reviewer (AA). To be included, studies had to evaluate an eReferral system for cost or outcome(s). Studies evaluating strictly e-consultation systems, studies not published in English, letters to the editor, and commentaries and opinions were excluded. Each document identified in the gray literature was checked against any publication reporting the same results. In that case, the journal article was included instead the gray literature document (for example, there was a report of e-referral program implementation in the San Francisco General Hospital that was published as an article,13). In a second step, two readers performed a full evaluation of the articles (AA and JET) for those studies selected at the first stage and were screened for final inclusion/exclusion. In case of differences in the selection for any article, both readers exposed their points and reached a consensus. References of the selected studies were also searched for inclusion/exclusion.
Data from these publications were extracted in a purpose-built data collection form. Key information abstracted included study characteristics (eg, country, year of study), study objectives, study design, and outcomes (cost or cost-effectiveness analysis; changes in quality of referrals, quantity, and efficiency of the eReferral process). Due to the nature of the studies, it was not possible to combine the studies using statistical methods (eg, meta-analyses). Rather, we described the studies on a narrative basis.
Results
Literature search results
The literature search, including the hand search of the reference lists of included papers, yielded 493 citations (377 from PubMed, 115 from OVID) of these, 205 articles were duplicates. After evaluation of titles and abstracts, we excluded 265 articles because they were not referring to e-Referral evaluations (see Figure 1). After a second screening of full texts (including the review update), eight publications were included in the review. In addition, we found one governmental report from the gray literature search that met the inclusion and exclusion criteria that also was included in the analysis. Four studies were conducted in the United States, two in Canada, one in the UK, one in Denmark, and one in New Zealand. The studies were published from 2004 to 2018. The majority of the studies (7/9) evaluated eReferral from primary care physicians (PCP) to hospital-based specialists, while the rest focused on PCP to non-hospital-based specialists.
Study design
Most studies (6/9) evaluated changes in referral patterns with the implementation of an eReferral system using surveys, questionnaires, and time-motion approaches. Only one study was a full economic evaluation that evaluated both costs (eg, handling or process costs, operational costs, and equipment costs) and benefits associated with the eReferral system. The remaining studies evaluated either the number of referrals, time to book an appointment, referral processing times, and provider satisfaction associated with eReferrals. While almost all studies reported the results using basic descriptive statistics (7/9), two studies conducted a regression analysis to estimate the impact of the eReferral implementation on the number of referrals. The study samples ranged from only a few referrals to those with more than 700,000 referrals. Tables 2 and 3 present more details on the characteristics of each study.
Table 2.
First author | Country | Objective | Study design | Sample characteristics | Referrals from–to | Type of outcomes | |||
---|---|---|---|---|---|---|---|---|---|
Cost or economic evaluation | Changes in quantity of referrals | Changes in quality of referrals | Changes in workflow efficiency or productivity | ||||||
Cannaby9 (2004) | Denmark | To evaluate the cost-benefit of electronic referral system | Time-motion | 41,235 referrals | General Practitioner to Hospital | YES | - | - | - |
Yeuen Kim10 (2009) | US | To assess the impact of electronic referrals on workflow and clinical care. | Web-based questionnaire | 298 General Practitioners | General Practitioner to Hospital | - | - | - | YES |
Scheibe11 (2015) | US | To evaluate the use and impact of an electronic referral system in rheumatology | 2,105 referrals | General Practitioner to Hospital | - | - | - | YES | |
Scott12 (2009) | UK | To evaluate time to process a referral | Interviews | 99 referrals | General Practitioner to Hospital | - | - | - | YES |
Kim-Hwang13 (2010) | US | To determine the impact of electronic referral on specialty referrals | Visit-based questionnaire | 505 questionnaires | General Practitioner to Hospital | - | - | YES | - |
Corwin14 (2014) | New Zealand | To assess whether feedback improves the quality of referral letters from general practice to secondary care and how electronic referrals affect the quality of referral letters. | Referral letters’ Feedback from peers | 340 referral letters | General Practitioner to Hospital | - | - | YES | - |
Chambers15 (2015) | US | To facilitate and sustain patient referral using an Electronic Health Record and a provider education program | Pre-post intervention analysis | 75 referrals | General Practitioner to Diabetes program | - | YES | - | - |
MacGregor16 (2009 | Canada | To evaluate the completeness and adequate number of referrals using an electronic referral system | Pre-post intervention analysis | 10 clinics (referrals number was not reported) | General Practitioner to Pediatric Hospital | - | YES | - | - |
Doumouras17 (2018) | Canada | To determine the impact on referral rates to bariatric surgery after converting to an online referral system from a fax-based system. | Pre-post intervention analysis | 5,317 referrals | All type of physicians or specialists to the Ontario Bariatric Network | - | YES | - | - |
Table 3.
Author | Main findings | Authors’ conclusion |
---|---|---|
Cannaby9 (2004) |
|
|
Yeuen Kim10 (2009) |
|
|
Scheibe11 (2015) |
|
|
Scott12 (2009) |
|
|
Kim-Hwang13 (2010) |
|
|
Corwin14 (2014) |
|
|
Chambers15 (2015) |
|
|
MacGregor16 (2009 |
|
|
Doumouras17 (2018) |
|
|
Economic evaluation of the eReferral systems
Only one study (1/9) was a full economic evaluation comparing the costs and outcomes associated with eReferrals in a cost-benefit analysis. In this study, Cannaby et al (2004) compared an eReferral system with a paper-based referral process in Denmark to estimate the time and resource utilization during the referral process between primary care providers to hospital-based specialists.9
Each stage of the referral process was mapped and then a time-motion approach was implemented to estimate the costs of processing a referral (eg, minutes to process a referral, number of times that the referral is returned with missing information, etc.). Staff costs were estimated for the eReferral system and the paper-based referral process by applying average wage rates to the time-motion data. Compared to the paper-based referral process, the eReferral system was estimated to save up to €1.6 in the average cost per minute spent during the referral process. The authors extrapolated these results to more than 700 thousand referrals in Denmark and found possible savings of €3,512,146 associated with the implementation of this eReferral system in Denmark.
Effects on efficiency of the referral process
Most of the studies analyzed changes in the referral process (7/9) (eg, workflow, wait times, and clinical care). Using an 18-item questionnaire, Kim et al (2009) conducted a study in the United States to assess the impact of eReferral on the workflow and clinical care on 298 primary care providers. The results indicated that 72% of the primary care providers felt that eReferral had improved the ability to track referrals, guide pre-visit work-up, and decrease wait times for new appointments. Overall 21% of the respondents reported no change in the workflow and clinical care while 7% were dissatisfied with the eReferral system.10
In another US study, Scheibe et al (2015) analyzed 2,105 referrals to rheumatologists from primary care providers from 2008 to 2012. They found that the main benefit of the eReferral system was the ability of specialists to participate in back and forth pre-consultation communication, including requesting more information. Their analyses suggested that pre-consultation exchange increased from 55% in 2008 to 74% in 2011. At the same time, only 63% of these pre-consultations were subsequently scheduled for a face-to-face appointment. The remaining 37% of the pre-consultations did not lead to a face to face appointment because the referrals were deemed inappropriate for a rheumatologist, the clinical reason leading to the referral was resolved by using the eReferral system or more information was requested before proceeding with an appointment.11
Scott (2009) conducted a study in Wales evaluating urgent cancer referrals from General Practitioners (GPs) to specialists through an eReferral system. In this study, 99 referrals from 5 GP practices were analyzed from January to July 2007 as part of a pilot program. The results indicated that the eReferral system improved the speed with which referrals were sent to the central office as well as the speed in which they were processed once received. Overall 81% of the 99 referrals were processed within 1 hr. In addition, GPs were highly satisfied with the eReferral system as measured by a 10-point likert scale (1= completely dissatisfied, 10= completely satisfied). The lowest satisfaction level recorded was an 8.12
Studies focus on the efficiency showed a reduction on wait times as well as inappropriate referrals. Additionally, they found a good perception from the care providers given a better track and communication with the patients.
Effects of electronic referral process on quality of referrals
Kim-Hwang et al (2010) conducted an analysis of the quality of referrals in the US associated with eReferrals compared to a paper-based system based on 505 questionnaires.13 Results indicated that the reason for referral was difficult to identify in 19.8% of the medical visits using a paper-based referral compared with only 11% of the eReferrals (p=0.03). Only 2.6% of the eReferrals were deemed not appropriate compared with 6.4% of the paper-based system, but this difference was not statistically significant (p=0.21). Regarding the need to follow-up, results varied depending on the type of provider. For example, medical specialty clinicians indicated that they requested follow-up in 84% of the paper-based referrals vs 90% of the eReferrals (p=0.06) while surgical specialists indicated that they requested follow-up in 76.2% of their paper-based referrals vs 58.1% of their eReferrals.
In a pre-post analysis performed in New Zealand, Corwin et al (2014) assessed two interventions to improve the quality of referrals and the percentage of adequate referrals to specialists.14 The first intervention evaluated whether feedback to GP (N=15) and nurses (N=2) from peers improved the quality of the referral letters to specialists. The second intervention evaluated the implementation of an eReferral system. Findings showed that the first intervention (ie, feedback to GP) improved the proportion of adequate referrals from 81.4% to 86.9% (p-value of 0.05). In the case of the eReferral system, the intervention did not improve the proportion of adequate referrals or quality of referrals. (p-value of 0.20).
Effects of electronic referrals process on quantity of referrals
The study performed by Chambers et al (2015) evaluated the changes in the number of diabetes referrals following the implementation of an eReferral system with and without training on the use of the system.15 The analyses were based on 75 referrals from 6 different health centers that occurred from April 2012 to November 2014. After the implementation of the eReferral system without training, the monthly number of referrals increased from 1 to 3. Following the implementation, training was offered and the average monthly number of referrals went from 3 to 6. Regression results confirmed that the trend in referrals before and after the implementation of the eReferral system was not significant while the trend in referrals increased significantly following the training of providers to use the electronic system.
A second study conducted by MacGregor et al (2009) evaluated the implementation of an eReferral system in a pediatric hospital in Canada. Before the eReferral implementation, 30% of the referrals were either incomplete or inappropriate.16 The implementation of the eReferral system resulted in a decrease of 2.7% of the incomplete or inappropriate referrals, as well as the increase of 6% in the total number of referrals processed.
Another Canadian study conducted by Doumouras et al (2017) examined the change in the number of referrals following the implementation of an eReferral system to replace the fax-based referral process for bariatric surgery in Ontario.17 Using regression analyses, the authors reported that the average number of referrals per year went from 244 in 2011 to 671 in 2015 due to the implementation of the eReferral system. Regression analyses adjusting for health regions, rural or urban, distance from the bariatric center, type and gender of physician and years of experience showed that overall 68% of the physicians increased their number of referrals after system implementation. The authors also examined factors associated with increase in eReferral. At the individual level, being a female practitioner, a primary care specialist and with 0–5 years of experience were associated with an increase of referrals after the eReferral system implementation. At the region level, living closer to Bariatric centers (<50 km) was also a statistically significant factor explaining an increase in electronic referrals.
The three studies found an increase in referrals through the implementation of eReferrals.
Discussion
Our review indicates that there is a great deal of variation in study design, setting, and measurement which limits the generalizability of the existing findings. Most of the studies conducted evaluations through the use of surveys or questionnaires. Only one study was a full economic evaluation (cost-benefit) making difficult to generalize or evaluate if e-referral process is cost-effective across different settings. Only two studies (2/9) performed an econometric analysis and the other seven used descriptive statistic analysis.
The lack of a standardized design of studies (for example, use of non-standardized surveys) may suggest the need for methodological standards and frameworks to evaluate eReferral systems across settings.
Nonetheless, the evidence presented in these studies collectively indicates that eReferral systems have the potential to improve the quantity and quality of referrals and are associated with a high degree of satisfaction by physicians in many settings. Studies showed the since the implementation of eReferrals, the number of referrals increased, but there is also a possibility of cost savings for the system, a reduction of unnecessary referral or follow-up, given a better idea of consultation as well as a good perception from PCP in quality and access. Findings also showed that e-referrals can reduce missing or incomplete information in the referral process avoiding back and forth between PCP leading to an improvement in wait times in consultation follow-up and less time to process referrals from providers.13
The main strength of our study was the systematic capture and review of nine studies evaluating eReferral systems in a variety of settings. We also defined the studies across logical categories that identify the main purpose and limitations to their evaluation. These categories allow decision-makers and researchers to distinguish the evidence-base and develop more complete study designs. However, further analysis should focus on the creation of feasible methodological standards for the design of economic evaluations in this area and measurement of a complete set of outcomes.
There are some limitations associated with our review. First, due to the small quantity of evaluations of eReferral systems, we were only able to describe the results and key findings of each study rather than conduct a more conclusive meta-analysis, regression analysis or related quantitative method to evaluate the overall effect of eReferral systems in quality, quantity, efficiency, and cost-effectiveness. Although we took a very systematic approach in searching the published literature, we did not keep track of the number of websites and documents that we reviewed for inclusion/exclusion. Similarity to the published literature, while we identified many potential documents, those were not related to e-referrals. Nonetheless like any literature search, it is possible that some studies are not captured in this review. Another limitation was that e-consultation and eReferral systems are described using similar terms, which made it more difficult to identify and delineate the evidence for eReferral systems alone.
Despite these limitations, the results of this scoping review indicate that further research is warranted to understand the effectiveness of eReferral systems within health care. Further economic analyses are needed to determine if initial capital and training costs are justified by efficiencies and health system benefits after implementation, and how referral volumes influence cost recovery. In addition, cost-analyses of e-referrals systems can also inform future cost-effectiveness or cost-benefit analyses. Nonetheless to better inform decision-making, comparative analyses based on strong study designs (eg, randomized controlled trial) are also needed to determine the effectiveness or cost-effectiveness of eReferral systems compared to self-referral systems and complete e-consultation systems in settings where more than one solution is viable.
We found that the evidence base for electronic referral systems is positive but limited. The evidence base is too limited to determine if eReferral systems are cost-effective interventions as most of the studies have evaluated eReferral systems on the basis of referral quantity and quality without considering costs. This scoping review is a first step to understanding the evidence-base for eReferral system implementation and methodological frameworks for their evaluation.
Disclosure
The authors report no conflicts of interest in this work.
References
- 1.Barua B, Ren F. Waiting Your Turn: Wait Times for Health Care in Canada, 2016 Report. Fraser Institute; 2016. [Google Scholar]
- 2.Stokes E, Somerville R. 2008. The economic costs of wait times in Canada, Report. The Centre for Spatial Economics. Available from: https://www.cma.ca/Assets/assets-library/document/en/advocacy/EconomicReport-e.pdf. Accessed July19, 2016.
- 3.Canadian Institute for Health information. 2012. Health Care in Canada, 2012: A Focus on Wait Times, Report. Available from: https://secure.cihi.ca/free_products/HCIC2012-FullReport-ENweb.pdf. Accessed September10, 2017.
- 4.Hughes C, Allen P, Bentley M. e-Referrals: why we are still faxing. Aust Fam Physician. 2018;47(1–2)50–57. [DOI] [PubMed] [Google Scholar]
- 5.Allison B, Shannon E, Valerie W, et al. Improving patient access to medical services: preventing the patient from being lost in translation. Healthc Q. 2009;13:61–68. [DOI] [PubMed] [Google Scholar]
- 6.Liddy C, Hogel M, Blazkho V, et al. The current state of electronic consultation and electronic referral systems in Canada: an environmental scan. Global Telehealth. 2015;209:75–83. [PubMed] [Google Scholar]
- 7. Naserias M, Adham D, Janati A J. E-referral solutions: successful experiences, key features and challenges – a systematic review. Mater Sociomed. 2015;27(3):195–199. doi: 10.5455/msm.2015.27.195-199 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Heimly V. Electronic referrals in healthcare: a review Stud Health Technol Inform2009;150:327–331. [PubMed] [Google Scholar]
- 9.Cannaby S, Wanscher C, Pedersen C, et al. The cost benefit of electronic patient referrals in Denmark full report. MedCom Available from: https://www.medcom.dk/media/1173/the-cost-benefit-of-electronic-patient-referrals-in-denmark-full-report.pdf. Accessed January15, 2017. [PubMed]
- 10.Yeuen K, Chen A, Keith E, et al. Not perfect, but better: primary care providers’ experiences with electronic referrals in a safety net health system. J Gen Intern Med. 2009;24(5):614–619. doi: 10.1007/s11606-009-1087-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Scheibe M, Imboden J, Schmajuk G. Efficiency gains for rheumatology consultation using a novel electronic referral system in a safety net health setting. Arthritis Care Res (Hoboken). 2015;67(8):1158–1163. doi: 10.1002/acr.22559 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Scott E. The Swansea electronic referrals project. J Telemed Telecare. 2009;15:156–158. doi: 10.1258/jtt.2009.003014 [DOI] [PubMed] [Google Scholar]
- 13.Kim-Hwang J, Chen A, Bell S, et al. Evaluating electronic referrals for specialty care at a public hospital. J Gen Intern Med. 2010;25(10):1123–1128. doi: 10.1007/s11606-010-1402-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Corwin P, Bolter T. The effects of audit and feedback and electronic referrals on the quality of primary care referral letters. J Prim Health Care. 2014;6(4):324–327. doi: 10.1071/HC14324 [DOI] [PubMed] [Google Scholar]
- 15.Chambers E, Wylie-Rosset BA. Increasing referrals to a YMCA-based diabetes prevention program: effects of electronic referral system modification and provider education in federally qualified health centers. Prev Chronic Dis. 2015;12:150–294. doi: 10.5888/pcd12.150294 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.MacGregor D, Parker S, MacMillan S. Innovation in managing the referral process at a Canadian pediatric hospital. Healthc Q. 2009;12(3):73–79. [DOI] [PubMed] [Google Scholar]
- 17.Doumouras, AG, Anvari S, Breau R, Anvari M, Hong D, Gmora S The effect of an online referral system on referrals to bariatric surgery. Surg Endosc. 2017;31:5127–5134. doi: 10.1007/s00464-016-5054-z [DOI] [PubMed] [Google Scholar]