Implementation Science Workshop: Implementation of an Electronic Referral System in a Large Academic Medical Center

CASE

Introduction

From 1999 to 2009, US specialty referral volume roughly doubled, and by 2009, 16.6 % of all hospital-based outpatient visits resulted in a specialty referral.¹ Despite this rapid growth, there are many problems with the specialty referral process. The process of making a referral is often not consciously constructed to serve the needs of primary care physicians or patients.² Deficiencies in timeliness of referrals, specialist access and communication are common. Additionally, primary care physicians (PCPs) cannot easily track whether referred patients had an appointment with the specialist. All of these problems undermine the quality of patient care and represent a malpractice risk.³ Improving the referral process by implementing an electronic referral ordering system could represent “low-hanging fruit” for quality improvement across many health systems.⁴

The literature on electronic referrals has encompassed two different types of systems: electronic referral order systems, which replace traditional phone- or fax-based methods of requesting referrals,^5,6 and electronic “curbside” systems, which are an asynchronous form of specialty consultation by electronic message.^7,8 In this project, we focus on electronic referral order systems, which have been established in the US and internationally, with broad uptake by PCPs and improved physician satisfaction.^5,6,9–12 However, these systems have been plagued by the same communication challenges as those observed with the traditional referral process, including ambiguous responsibility for follow-up and unclear reasons for referral.^10,11

At our institution, Brigham and Women’s Hospital (BWH), the specialty referral process was problematic as well, with 63 % of PCPs indicating dissatisfaction with the procedures that were in place.¹³ Long wait times for new visits to many specialties forced BWH PCPs to refer to non-BWH specialists. These non-BWH specialty visits increase fragmentation of care and hinder coordination of specialty care for our primary care population. As BWH participates in more shared savings contracts, including Medicare’s pioneer accountable care organization (ACO) initiative and the Blue Cross Blue Shield of Massachusetts Alternative Quality Contract, BWH has greater financial incentives to reduce fragmentation in care and to keep specialty referrals within our system.^14,15 These incentives are further heightened by a new state law that creates a cap on health care cost growth for all health systems.¹⁶

After identifying the specialty referral process as a quality improvement target, BWH set out to design and implement an electronic specialty referral ordering system (“eReferral”) to address these problems. In this evaluation, we sought to add to the existing literature on eReferral systems by tracking physician satisfaction across an entire health system and using administrative claims to observe the impact of eReferral on total specialist utilization.

Setting and Participants

BWH is an academic medical center with ~135,000 primary care patients seen by 162 PCPs at 14 clinics. In 2013, there were ~247,000 PCP and ~105,000 new specialist visits at BWH. Of new specialty visits, 30 % were from BWH primary care patients. The new eReferral system has been implemented across all BWH PCP clinics.

Program Description

In 2012, we performed a qualitative process mapping of our referral process in collaboration with PCPs in our system using a Lean framework, and found several systematic problems (Table 1).¹⁷ There was frequently no communication between the specialist and PCP before initial consults, and there was no formal mechanism to triage referrals as urgent or non-urgent if necessary. Many referrals were also dependent on personal relationships (e.g., e-mailing a surgical colleague to see a patient) that made it difficult for new PCPs to navigate the system of specialists. Lastly, the referral process was too complicated. A “process map” of the referral process found that a typical primary care referral involved 12 steps, with 17 variations, between up to five different parties (Fig. 1a). Our electronic referral system was therefore designed to improve communication and to both simplify and standardize the referral process (Fig. 1b).

Table 1.

Example Deficiencies Identified in the Referral Process at BWH

Problem	Description	How eReferral addresses problem	Evaluation metrics
Difficult to navigate system of specialists	Referring physicians—both PCPs and specialists, both internal and external—have to re-learn referral processes for each specialty, and sometimes for each physician	Standardizes referral process, allowing for referral to specific specialist or general referral	PCP-reported referral satisfaction
			PCP uptake of eReferral
Referral tracking	PCPs have no system to track whether essential referrals are completed	Automatic electronic alerts created if referrals are not scheduled or completed	PCP-reported confidence in referral tracking
Referral system based highly on personal relationships	Experienced physicians e-mail trusted colleagues to book urgent appointments and for advice and referrals, while newer physicians may struggle	Makes it easier for PCPs to make general specialty referrals and let the specialists find the right physician	PCP survey-specific physician referral preference
			Percentage of eReferrals to specific physicians
Poor systems to effectively triage referral requests to patient need	PCPs often feel that it was difficult to get their patients seen in a timely fashion for most of the frequently referred specialties	Includes mechanism for PCP to indicate urgency	Time to appointment
			Distribution of referral triage requests
			PCP perception of access
Fragmentation of specialist care outside of BWH	Substantial proportion of visits to specialists for patients with BWH PCPs happen outside of BWH, which complicates system-wide quality improvement activity	Makes the process of referring within the system much easier, thus encouraging within-system referrals	Percentage of new specialist visits for BWH PCP patients outside BWH

Fig. 1.

a Referral workflow diagramed via PCP focus group interviews prior to implementation of eReferral. b Our concept of a reformed referral process using information technology. Abbreviations: LMR longitudinal medical record, the BWH electronic medical record system, IDX BWH electronic scheduling system, MD medical doctor, RM referral manager, a basic referral request system within the BWH electronic medical record system

Creating “Buy-In” with Physicians and Executive Leadership

The eReferral system touches almost every BWH specialty and impacts the entire office, from physician to front desk staff. Our first objective was building engagement and “buy-in” on the need for change among primary care and specialty departments. First, using the “medical neighborhood” definition published by the American College of Physicians,¹⁸ we created a collaborative care agreement that defined the expectations of primary care and specialist physicians around shared patient care.¹⁹ The agreement had two purposes: first, it outlined a set of principles to which all physicians could be held accountable, and second, it defined what features were necessary in the eReferral tool.

To make the case to hospital leadership that this was an important clinical issue, we analyzed health plan data and found that almost 40 % of BWH PCP referrals were going to non-BWH specialists (see methods discussed below). These outside system specialist visits make it harder to coordinate specialist care, and can lead to duplicative care. Pilot survey data from our PCPs and specialists in late 2012 (before this study) showed that poor access and communication were important contributors to out-of-system referrals. These results resonated with hospital leadership, and helped us garner their support for eReferral.

Based on prior research and our pilot work, we had five key goals for the eReferral system: 1) increase PCP satisfaction with the referral process, 2) improve referral tracking, 3) reduce the need for personal relationships to drive referrals, 4) enable effective triage of referral urgency to alleviate access problems, and 5) reduce referrals outside the BWH system. We established practical metrics that could best assess the impact of eReferral on these key goals, shown in the “evaluation metrics” column in Table 1.

Implementing eReferral

We contracted with an outside vendor, Par8o, (https://www.par8o.com/) to integrate the eReferral system into our electronic medical records (EMR). The focus of eReferral was simplicity and standardization. The eReferral system automatically pulls patient demographic information from the EMR. On the first screen, the referring provider chooses the desired specialty for a referral, and on the second screen, enters a brief reason for referral and selects an urgency level (urgent, 3 days or less to patient convenience). Physicians can select a specific specialist or refer to the first available specialist in a group (e.g., electrophysiology or general cardiology). The referral is then sent to an electronic queue managed by the specialty department’s administrative staff, who will call the patient within one business day, on average, to book an appointment. This step bypasses the need for the primary care office to serve as an intermediary in scheduling, a step that was often inefficient and frustrating for both staff and patient, or for the patient to call and schedule on their own. The eReferral program was rolled out during January 2014 across all primary care clinics. The rollout was accompanied by extensive training by the BWH eReferral team, which included an hour-long session and onsite support during the first 2 days of the launch for all PCP practices, and five hour-long training sessions at a central site for all specialist practices.

Program Evaluation

Our evaluation of eReferral was conducted as a prospective cohort study of all BWH PCPs and primary care population. We measured physician outcomes via survey and patient outcomes using data from billing claims and scheduling databases. This study was deemed exempt from review by the Partners institutional review board.

We measured physician outcomes by surveying the 162 PCPs practicing at least one session a week. Our primary physician outcome was overall PCP satisfaction with the referral process. Secondary outcomes included confidence in referral tracking and perceived access to the ten common specialties (full survey in Supplemental Appendix). The two waves of survey were administered in December 2013 (1 month before eReferral adoption) and September 2014 (9 months after eReferral rollout). Physicians received invitations via e-mail and completed the survey using a Web-based interface (SurveyMonkey, Palo Alto, CA, www.surveymonkey.com). We received 105 (65 % response rate) and 102 (63 %) complete responses for the pre- and post-eReferral surveys, respectively.

We measured three main patient outcomes: time from request to scheduled appointment, no-show rates, and total new specialist visits. These were tracked monthly from August 2013 to August 2014 for both BWH primary care patients and all BWH patients. Using BWH scheduling data, we tracked no-show rates and time from request to scheduled appointment (categorized into intervals of 0–3 days, 4–7 days, 8–30 days and 31+ days). Using BWH billing data, we captured new specialist visits for all BWH patients, including BWH primary care patients. Using data extracted from insurance claims for 24,552 BWH primary care patients (~20 % of all BWH primary care patients) covered by the three largest commercial payers, we calculated the total number of new specialist visits to BWH and non-BWH specialists to calculate the rate of out-of-system specialist referrals. Using the eReferral system, we supplemented the three main outcomes by tracking the overall number of eReferral requests by triage urgency category and group vs. specific physician referral designation.

We used z-tests for proportions to test the difference in PCP survey results before and after eReferral implementation. To test for trends in visit volume, we used linear regression to estimate the coefficient for month (from August 2013 to August 2014) as the independent variable, with visit volume as the dependent outcome. All analyses were performed in R (v3.0.1, R Project for Statistical Computing). We considered a two-sided p value of <0.05 as statistically significant.

RESULTS

The eReferral process was rapidly adopted by PCPs: by August 2014, 93 % of all new specialist visits for the BWH primary care population were made via eReferral. Adoption of eReferral was accompanied by an increase in new BWH specialty visits among BWH primary care patients (p = 0.001 for trend). This increase was not seen for non-BWH PCP patients (p = 0.26 for trend) (Fig. 2). Among the ~20 % of BWH primary care patients in shared savings contracts, monthly new specialist visits to all specialists (BWH and non-BWH) increased from 2.35 to 2.82 new specialist visits per 100 patients per month from August 2013 to July 2014 (p = 0.007 for trend; Fig. 3). There was no significant change in the volume of new specialist visits outside BWH from August 2013 to July 2014 (p = 0.33; Fig. 3).

Fig. 2.

Overall monthly volume of new specialist visits and adoption of eReferral from August 2013 until August 2014. The solid line shows monthly new specialist visits for the BWH PCP population, which is the population of patients who have assigned PCPs in the BWH system, and the dotted line shows monthly new specialist visits for the BWH non-PCP population, which represents all other patients seen at BWH who do not have an assigned BWH PCP. The dashed line shows monthly volume of eReferral requests. The eReferral program started in January 2014 (vertical line). New specialist visit data from BWH scheduling database (IDX), eReferral requests from eReferral database. p value for positive trend in new specialist visits for BWH non-PCP population, p = 0.26 and for PCP population p = 0.001. *eReferral requests exceed total number of BWH PCP population new specialist visits due to the combination of declined referrals after referral made and no-show appointments

Fig. 3.

Total new specialist visit volume within and outside BWH for the commercial shared savings contract population with a BWH PCP (approximately 26,000 patients). The eReferral program started in January 2014 (shaded area). p values for trends from August 2013 to July 2014: p = 0.007 for all new specialist visits per 100 patients, p = 0.004 for BWH specialist visits and p = 0.33 for new specialist visits outside BWH

With the use of eReferral. PCP-requested triage urgency was heavily skewed towards non-urgent referrals: only 19 % of referrals were 1 week or less, 36 % were 1 month or less, and 45 % of requests were “patient convenience.” Specialty offices met the PCP’s requested triage time for 72 % of referrals within 3 days (most urgent category).

Tracking of time between referral and appointment before and after eReferral implementation showed that appointments 31 days or more increased modestly, from 33 % to 34 %, after eReferral implementation (p = 0.07 for difference), and appointments 7 days or less decreased from 26 % to 24 % (p = 0.003 for difference) (Appendix Figure). There was no change in the no-show rate (average monthly rate 11.6 %) throughout the period of this study (p = 0.65, data not shown).

The percentage of PCPs agreeing or strongly agreeing that they were satisfied with the overall referral process increased from 21 % to 69 % after implementation (p < 0.001; Table 2) The percentage of PCPs agreeing or strongly agreeing that they could confidently track referrals increased from 20 % to 45 % (p < 0.001). PCPs reported an increased ability to obtain timely access to five of the ten common specialties (Table 3).

Table 2.

PCP Survey Responses

	Pre-eReferral (n = 105)	Post-eReferral (n = 102)	p value*
Satisfied with referral process	21 %	69 %	<0.001
Confident in referral tracking	20 %	45 %	<0.001
Prefers referring to specific MD	55 %	34 %	<0.003

Values represent proportion of PCPs selecting “Agree” or “Strongly Agree” with survey questions about overall referral satisfaction, confidence in referral tracking and preference for referring to specific physicians. See Supplemental Appendix for survey questions

*p values calculated with z-test for proportions

Table 3.

PCP Perceptions of Timely Access to Specialist Care

Specialty	Pre-eReferral (n = 105)	Post-eReferral (n = 102)	Difference	p value*
Cardiology	79 %	80 %	1 %	0.81
General Surgery	69 %	86 %	18 %	0.003
Rheumatology	68 %	80 %	13 %	0.04
Dermatology	64 %	72 %	8 %	0.24
Orthopedics	63 %	83 %	20 %	0.001
Otolaryngology	60 %	74 %	14 %	0.04
Ob/Gyn	56 %	61 %	5 %	0.50
Endocrinology	48 %	53 %	5 %	0.45
Gastroenterology	37 %	53 %	16 %	0.02
Neurology	21 %	28 %	7 %	0.21

PCP perceptions are measured by the proportion of PCPs who agree or strongly agree with the statement that “My office staff can get a timely appointment for our patients when needed” for specialists in the categories above. Proportion of PCPs indicating that they agree/strongly agree are shown for surveys before and after implementation of eReferral

*p values estimated using z-test for proportions

The proportion of PCPs who agreed or strongly agreed that they preferred to refer to specific physicians decreased from 55 % to 34 % (p = 0.003). Only 19 % of all eReferral requests were for a specific physician.

Challenges and Future Plans

Based on our goals and these preliminary findings, we consider the eReferral implementation to be a success, with a few key caveats. First, we found that PCP overall satisfaction with the referral process increased dramatically, from 21 % to 69 %. There was also a twofold increase in the percentage of PCPs who had confidence in their ability to track referrals (though a majority of PCPs still did not indicate high confidence). PCP perceptions of access also increased for half of the specialties on the survey. An unexpectedly large proportion of undifferentiated referrals (a specific physician not indicated) made it easier to quickly schedule appointments with the next available specialist. This was unanticipated, since in our pre-eReferral survey, a majority of PCPs indicated a preference to refer to a specific specialist (Table 3). One explanation for this finding could be related to the user interface for eReferral: it requires only one click to make a general department referral, as opposed to the extra seconds it takes to type in and search for a specialist.

However, there are several key caveats. Even though PCP perceptions of access improved markedly, we observed little change in actual access times (time from referral to appointment)—which, if anything, became longer. Given the large percentage of non-urgent referrals (1 month or more), one explanation for this discrepancy could be the better “sorting” of referrals into non-urgent and urgent. Pre-eReferral, there was no easy mechanism to allow a PCP to distinguish between a patient who could be seen in 2 months versus a patient who should be seen within a week. The improved perceptions of access could therefore be capturing PCP’s improved control over triage urgency.

In addition, we did not observe a significant change in the volume of out-of-system referrals. Reducing out-of-system referrals was an important focus for BWH leadership, and eReferral was only one aspect of the organization’s efforts to reduce this practice. It is possible that over a longer period of time, the use of eReferral will lead to a decrease in referrals outside BWH, which will require a longer post-evaluation period. Globally, total specialist visits increased among our primary care patients in shared savings contracts. Because we measured specialist visits using insurance claims, we were able to capture patients' total specialist utilization, which revealed that the increase in specialist visits was driven primarily by an increase in specialist utilization within BWH. This overall growth in the utilization of specialists likely reflects the new referral system’s ease of use, which reduces barriers to referral.

Our findings add to the growing body of literature on electronic referral systems, showing that high satisfaction with a new eReferral system can be accompanied by an increase in referral rates. For health systems participating in global payment or shared savings contracts, this increased rate of new specialist visits could mitigate the benefit of eReferral due to the potential for increased costs from higher specialist use. A possible solution for this problem could involve integrating formal electronic “curbsides” into an eReferral program which could potentially obviate the need for many referrals. Other electronic curbside programs have shown impressive decreases in the rate of specialist visits.^7,20

Limitations of this analysis include our focus on one institution as well as the use of proxy measures through physician surveys. For instance, we measured confidence in referral tracking, but did not measure consequences in terms of a change in the number of completed urgent referrals. We were also unable at this time to measure patient-level outcomes aside from referral volume, though we plan to address patient satisfaction in the future. Further analyses will include the impact on “harder” endpoints such as outpatient medical spending per patient, which we intend to evaluate in future work.

In conclusion, our implementation of an electronic referral order system demonstrated rapid adoption associated with marked improvements in PCP satisfaction and confidence in the referral process. This adoption was faster than we would have expected based on prior studies.^21,22 However, eReferral was also accompanied by an increase in the total number of new specialist visits per patient among the BWH primary care population. Further study of eReferral across a wider diversity of health systems will help to quantify the global impact of such systems on the specialty referral process.

TEACHING COMMENTARY

By Hardeep Singh, MD MPH

This study evaluates an electronic referral ordering system designed and implemented at a large academic center in order to address problems with timeliness, specialist access and communication. Evaluation metrics tracked physician satisfaction across the health system and used administrative claims to observe the impact of eReferrals on total specialist utilization. The adoption of this system led to improvements in PCP satisfaction and an increase in new specialist visits per patient. In this commentary, I first discuss why the use of a theoretical framework is essential in implementation and evaluation research related to health information technology (IT), and then outline key teaching points using examples from this case study.

Although the implementation of the eReferral system in this case was largely considered a success, not all health IT-based innovations meet a similar fate or lead to an expected improvement in quality and safety.²³ The success of eReferrals depends in part on effective and timely communication to facilitate information sharing and transfer of patient care responsibilities between outpatient providers. However, even when supported by technology, referral communication between PCPs and specialists can be less than satisfactory²⁴ and in the past has led to new communication challenges and unintended consequences, often related to non-technical factors.^11,25 Thus, methodological approaches for implementing and evaluating outcomes of health IT-related communication interventions need to address both technological (software/hardware) and non-technological “social” (e.g., organizational components, people and workflow)²⁶ contextual factors. These approaches, for instance, must be grounded in an understanding of how referral processes fit within the complex “sociotechnical” context of health IT-enabled care.^25,27

Most existing implementation frameworks are not sufficient to specifically evaluate health IT-based quality improvement or patient safety-related interventions. To effectively assess health IT-based interventions, frameworks or approaches must consider technology within the larger sociotechnical system in which it is implemented. The Institute of Medicine proposes the application of concepts from human factors to the improvement of health care systems. ^28,29 For instance, human factors engineering-based approaches may address not only electronic health record (EHR) software and hardware features, but also organizational policies, user behaviors, and workflow issues that facilitate or hinder safe and effective communication. To ensure success in this case study, several aspects of workflow were addressed prior to the implementation of eReferral. Process mapping of the referral process using a Lean framework identified workflow-related problems beforehand. The eReferral system was then designed to overcome the communication barriers as well as to simplify and standardize the process. The system supported provider workflow by automating certain tedious or repetitive steps that previously required manual effort. All of these are examples of making technology fit well with the referral workflow to maximize chances of successful adoption and use.

In sum, the theoretical framework needs to account for the complex “sociotechnical” context of health IT-enabled care. Any evaluation must consider both technical and non-technical components related to the health IT intervention, as well as intended and unintended consequences of IT implementation.

Teaching Point 1: An Evaluation Framework for Health IT Must Account for the Numerous Challenges and Complexities Associated with Its Implementation and Use

In order to conduct a multifaceted evaluation that accounts for interactive and interdependent sociotechnical dimensions of health IT-enabled care,³⁰ we propose the use of a recently developed eight-dimensional sociotechnical model (Sittig and Singh’s sociotechnical model^30,31). This conceptual model is informed by Carayon’s Systems Engineering Initiative for Patient Safety (SEIPS),³² a human-factors engineering model that addresses work system design for patient safety.

The eight dimensions of the model include 1) hardware and software; 2) the clinical content within the hardware/software; 3) the user interface that allows clinicians to interact with technology applications; 4) the people who configure the IT system, train users and use the IT system, and patients; 5) the clinical workflow and communication processes that enable people to provide patient care; 6) the internal organizational features (policies, procedures, environment and culture) that “govern” all of the activities associated with using IT; 7) the external rules and regulations that affect the healthcare delivery system; and, finally, 8) the measurements and monitoring required to determine what is working and what is not within this IT-enabled clinical work system. Evaluation dimensions can often be anticipated by addressing the eight dimensions of the model.^33,34

Experience with several studies^31,35–37 has revealed that consideration of all eight dimensions of the sociotechnical model is essential to an understanding of successful implementation, use and evaluation of health IT-related interventions. Many of these dimensions were addressed in the case study, as detailed below. This holistic approach is critical, because inadequate attention to one or more of these dimensions can often explain why a health IT-based intervention was not successful. Mixed-methods approaches are usually needed to address the various model dimensions.

Teaching Point 2: Evaluation Must Determine Whether Technology is “Usable” and an “Enabler” of Clinical Work

Software or hardware (i.e. the computing infrastructure used to power, support and operate clinical applications and devices) is foundational to any health IT initiative. Just as was done in this case study, one might consider whether the software was well designed and standardized. For instance, electronic referral software that does not harness the benefits of EHR data to automatically pre-populate fields in the referral template when possible (e.g., demographic data, current medication list, recent relevant laboratory test results)³⁸ will be considered poorly designed, and will essentially increase the clinician's workload. The eReferral system automatically populated patient details from the EHR, thus making it easier to use. Rigorous formative usability testing before the software is implemented³⁹ or post-implementation usability tests in the "real world" could be helpful in this regard.

The content of a referral letter greatly influences the referral process.^40–44 Much of eReferral software "content" was standardized, and allowed the referring provider to choose the specialty, enter a brief reason for referral, and select the urgency of the referral.⁴⁵ Studies have shown increased physician satisfaction and better feedback from specialists when referral templates are used to standardize referral communication.^43,46,47 For instance, capturing the reason for consult is a key but often omitted step in the referral process,^13,48–51 and thus electronic referrals must be designed to include a clearly defined justification of the referral. Furthermore, communicating details related to urgency is an important component of referral content.⁴⁵ Prior to eReferral implementation, there was no easy way for providers to indicate which patients needed a more rapid referral.

Most smartphone users can appreciate a well-designed user interface, and we should expect no less from health IT interventions. Referral template user interfaces should always be designed to minimize cognitive load on the provider making the referral.⁵² The user-friendly interface of eReferral required only one click to complete a general departmental referral, making scheduling appointments easier and quicker.

In sum, better outcomes can be expected when technology is “usable” and an “enabler” of clinical work.

Teaching Point 3: Always Consider the Impact of Internal or External Policies when Evaluating Health IT

Within institutions, lack of clear policies and procedures can result in several breakdowns in the referral process, leading to inefficiencies in patient care, provider dissatisfaction, and the potential for delays in diagnosis and treatment.¹¹ The eReferral implementation addressed organizational policies and procedures through the creation of a collaborative care agreement that defined both primary care (or referring) provider and specialist physician expectations and accountability about shared care. Collaborative efforts between referring providers and specialists to facilitate communication and clarify referral expectations not only reduce referral denials, but in essence are fundamental to achieving optimal outcomes from electronic referrals. In the case study, these efforts also helped define the features that were necessary in the eReferral tool. Similarly, external factors such as reimbursement systems, legal considerations, national quality measurement initiatives, accreditation and other policy and regulatory requirements all have influence on the success of health IT-based communication interventions.³³ The case study recognized these external rules and regulations by considering referral rates and their possible impact on health systems participating in global payment or shared savings contracts.

Teaching Point 4: Evaluation Must Consider Measuring Both Intended and Unintended Consequences of Health IT Implementation and Use

In evaluating electronic referrals, organizations may need to conduct several process measurements (e.g., completed referrals, no-shows/missed appointments and denied or cancelled referrals). Although the EHR facilitates transmission of useful information at the PCP–subspecialist interface, getting information from point A to point B is not always fail-safe. In a previous work, we found that at 30 days post an electronic referral, 6.3 % of referrals had an unexplained lack of follow-up action by subspecialists, and 7.4 % of discontinued referrals returned to PCPs were associated with an unexplained absence of follow-up.²⁵ Some issues must be proactively thought through prior to implementation. For example, with a few easy clicks, specialists could receive so many referrals for trivial problems that they wouldn't have time for the important ones, and this could create a huge backlog. Organizations could also predict some of the unintended consequences by conducting proactive risk assessments using the provider communication-related SAFER (Safety Assurance Factors for EHR Resilience) guide^53,54 endorsed by the Office of the National Coordinator for Health Information Technology (ONC).

Although physician satisfaction, no-show rates, access time and specialist utilization (including out-of-system referrals) were monitored in this study, the ultimate evaluation is the study of how health IT interventions lead to improved communication and improved patient outcomes. A four-phase iterative measurement approach could be used,^23,54 where phase I consists of measurements to show that the system is “available”, phase II shows system “use”, phase III shows the system’s effect on performance measures associated with IT use, and phase IV involves the measurement of unintended consequences.

In conclusion, evaluating health IT innovations such as electronic referrals requires a better understanding of how they fit within the complex “sociotechnical” context of health IT-enabled health care.^25,27 Improving communication remains a challenge, even in institutions that use comprehensive state-of-the-art health IT systems.²⁵ Rigorous evaluation must account for the complexities of health IT implementation and use, and must determine whether technology is usable and enables clinical work. In addition to considering non-technical factors, evaluation should measure both intended and unintended consequences. Sociotechnical frameworks are useful for informing methodologies that enable a comprehensive evaluation of health IT-based interventions, thus enhancing our understanding of successes and failures of health IT. Only through a sociotechnical lens can we improve the use of health IT as a means to serve our patients.

Electronic supplementary material

Below is the link to the electronic supplementary material.