Implementation of an Electronic Referral System for Outpatient Specialty Care

Abstract

Poor communication between primary care and specialists often leads to delays, inefficiencies and suboptimal patient outcomes. This study examined implementation of an electronic referral system (eReferral) that creates direct communication between primary care providers and specialist reviewers. Semi-structured interviews were conducted to assess factors affecting the success of eReferral implementation; transcripts were analyzed using qualitative methods. Primary and specialty care providers were enthusiastic about the system. Primary care providers had favorable attitudes despite a number of challenges including increased workload due to a shift in tasks from specialists and administrative personnel, poor connectivity, and insufficient hardware. System acceptance was driven by perceptions of improved access to specialty care, better appointment tracking, and improved communication between primary and specialty care providers. Synergy among development processes, implementation practices, and technical factors, including human-centered design, iterative development, a phased rollout, and an intuitive user interface, also fostered uptake of the system.

Introduction

Studies document a range of problems with the primary care-specialty care interface, including inappropriate and inefficient use of specialty care services,^{1, 2} unclear guidelines about the role of specialists,³ and poor communication between referring and specialty providers, characterized by a lack of information about the reason for the referral,^4–6 failure of the specialist to transmit findings back to the referring physician,^{2, 4, 6} and paper or fax-based forms that can be difficult to decipher and hard to track.^{7, 8} Health information technology (HIT), such as electronic health records (EHR), has the potential to improve referral processes by automating work processes and by providing a mechanism to track referral requests. Some evidence shows that electronic referral systems improve these processes.^8–10 However, most systems are used primarily to transmit referral requests, which are used to schedule patients on a first-available basis. Given the importance of communication between primary care providers (PCPs) and specialists for efficiency and quality of health care delivery,^{4, 11, 12} the University of California San Francisco (UCSF) in conjunction with San Francisco General Hospital (SFGH), San Francisco’s main safety net provider of specialty care, developed a web-based referral system (eReferral) that allows for iterative communication between referring and specialty providers.

The SFGH eReferral System

eReferral is a HIPAA-compliant web-based referral and consultation system that replaces hardcopy, telephone, and fax referral requests. The web-based application is tightly integrated into the hospital’s existing EHR such that users are not aware that eReferral is a separate program. For example, the application pulls data from existing provider and patient demographic and clinical databases to populate referrals. The eReferral process is depicted in Figure 1. The provider submits a referral request to the desired specialty service from within the EHR. Some specialties have screening questions that are designed to direct referring providers to the correct clinic, and all specialties have a policy page with information such as requirements for submitting a referral (e.g., completion of specific pre-visit work-up). The provider then enters the clinical consultative question in free text format. Relevant patient and provider information is automatically appended from the EHR, and the completed referral is sent to the specialty service’s electronic queue. Each specialty service has a designated clinician (physician or NP) who reviews and responds to each referral request. The specialist reviewer can use the eReferral system to communicate with the referring provider to address the patient’s issue, with or without an appointment. By channeling all incoming referrals for clinical review and creating a mechanism for iterative primary care–specialty communication, eReferral is designed to eliminate inappropriate referrals, identify and expedite urgent cases, and reduce premature referrals by ensuring proper and complete primary care work-up prior to scheduling an appointment with a specialty clinic.

Figure 1.

The eReferral Process

eReferral was developed by a team consisting of a gastroenterology (GI) specialist and SFGH IT staff. The system was first implemented for referrals to the SFGH GI clinic. SFGH/UCSF obtained subsequent funding for wider implementation of eReferral. The implementation team, which consisted of primary and specialty care providers, worked with the developers to roll out the system. The team conducted extensive outreach and education both to teach PCPs how to use the system and to solicit user feedback; for example, a PCP survey conducted in 2007–2008 generated 11 pages of comments that were systematically sorted and addressed. At the time of this study, the system was in use in 5 specialty clinics in addition to GI, and to date it has been implemented in more than 30 specialty clinics and services. After implementation within a given specialty, use of eReferral was mandatory.

Prior studies of eReferral included a survey of all PCPs who used the system in 2007–2008 and a visit-based survey of specialists who filled out questionnaires about specialty clinic visits in real time, both before and after the implementation of eReferral. These studies found high acceptability of eReferral among PCPs respondents, 71.9% of whom believed that eReferral improved overall clinical care.⁷ A large majority (89.0%) reported that eReferral (compared to paper based referrals) improved tracking of referrals and 73.0% believed it improved pre-visit work-ups. The survey of specialty visits showed that rates of specialists being unable to discern the reason for referral dropped significantly when comparing referrals made via eReferral (9.5% of surgical referrals) vs. 38.0% of surgical consult visits made with paper referrals. The use of eReferral also led to lower rates of follow-up visits for specialty surgical visits and lower rates of follow-up visits deemed to be avoidable (44.7% compared with 13.5% for surgical visits).¹³ The introduction of eReferral led to decreased wait times for new appointments, presumably by decreases in inappropriate referrals and avoidable follow-ups.¹⁴

This study sought to obtain a more nuanced understanding of the benefits and drawbacks of eReferral through semi-structured interviews of eReferral users, including clinical and administrative staff on both the referring and specialty. We also sought to understand the implementation process and factors that led to the successful uptake of the system.

Method

Study Population

Staff in a purposive sample of four primary care and three specialty clinics participated in the interviews. The four primary care sites included a hospital-based clinic, a Community Oriented Primary Care (COPC) clinic, and two San Francisco Community Clinic Consortium (SFCCC) clinics. The hospital-based and COPC clinics are part of the San Francisco Department of Public Health (SFDPH), the same organization that operates SFGH and the specialty clinics, whereas the SFCCC clinics are independent nonprofit entities. These SFCCC clinics must access the SFDPH EHR through a virtual private network connection, which is more cumbersome in that it requires using multiple logins and traversing two firewalls. All primary care practices had been using eReferral for 2–3 years in selected clinics at the time of this study, although new specialty clinics were added to eReferral over time, and there were continual enhancements to the system during the study period. Specialties included one clinic (nephrology) that had not implemented eReferral at baseline and two clinics that had previously implemented eReferral but that differed in patient populations and procedures. For example, cardiology is a relatively high volume clinic (88 referrals per month) in which treatment decisions are highly dependent on physical examination. Endocrinology has a lower volume (47 referrals per month) and specialists can often advise on treatment decisions based on their interpretation of laboratory results. At the time of the interviews, these specialty clinics had been using the system for 10–14 months.

The medical director at each site identified staff members who use eReferral or whose jobs were affected by the system as possible interview participants. There were a total of 23 participants in the study. In each primary care practice, we interviewed 2 clinicians (physicians or nurse practitioners) who used eReferral and one administrative staff member. In two of these sites we also interviewed the office manager to obtain higher-level perspectives on workflow changes. In each specialty clinic, we interviewed the physician-reviewer, one non-reviewing physician, and 3 staff members who scheduled specialty care visits. We also conducted one-year follow-up interviews of three PCPs and two specialists, focusing on their reactions to enhancements to eReferral in the intervening year.

Interview Protocol

The study protocol was approved by the RAND and UCSF IRBs, and all interviewees gave informed consent prior to participating. We conducted 60-minute interviews with PCPs and specialists, who were paid $100 for participating, and 30-minute interviews with administrative staff, who were paid $50 for participating. One or two interviewers conducted the interviews; three interviews were conducted face-to-face, and the rest were conducted by telephone. Interviews were audio-taped and transcribed. One recording was lost due to technical difficulties, yielding a total of 27 transcripts.

Interviews questions addressed users’ expectations about or perceptions of the effects of eReferral; perceived benefits of organizational innovations are an important aspect of successful technology adoption.^{15, 16} We assessed participants’ overall attitude toward eReferral and their views of the impact of the system on principal performance goals including clinical efficiencies, reduced wait time, and administrative efficiency. eReferral also has the potential to affect other processes and outcomes in primary and specialty care that may influence the business case for implementing such a system. Interview prompts addressed the effects of eReferral on processes such as changes in workload or work roles and communication and coordination. In addition, we assessed implementation practices, focusing on technical and organizational facilitators and barriers that are typically associated with the uptake of new information technologies. Following previous research,^15–18 we included questions regarding: (1) technical factors, particularly as related to system ease of use (e.g., user interface, network reliability); and (2) facilitating conditions, such as training, social influence, resources, and support. Assessment of organizational factors, in particular, is absent in many studies of the uptake of HIT.¹⁹

Analysis

We used the template organizing style of qualitative analysis.²⁰ In brief, this method involved first developing a set of codes to reflect the research topics under investigation. For example, we had general codes corresponding to categories such as performance gains, ease of use, workload/workflow, and implementation processes, with sub-categories reflecting specific instances of each. We also created categories for emerging themes. We applied these codes to sentences or paragraphs (units) in the interview transcripts using Atlas.ti software. The codes were then iteratively refined during a calibrating phase in which two coders (including the first author) coded five transcripts, and differences in codes were discussed and resolved. The first author subsequently unitized the remaining transcripts (i.e., divided respondents’ answers into logical passages), and both raters coded four additional transcripts independently. These four transcripts consisted of 109 units, and each unit could be coded into one or more of 40 sub-categories. Cohen’s kappa across all categories was .61, indicating a satisfactory level of interrater reliability beyond chance.²¹ Raters discussed and resolved disagreements on categories with lower levels of agreement. The first author coded the remaining 18 transcripts.

Results

The results consist of summaries of the main themes, with illustrative quotations from the interviews. Ellipses indicate text that was omitted from quotations to maintain respondent anonymity or to eliminate extraneous information.

Clinical and Administrative Processes and Outcomes

Overall attitudes toward eReferral

Most interview participants were either generally satisfied with eReferral, e.g., “… I’m pretty satisfied with it...I don’t hear a lot of negative feedback “ or extremely satisfied, e.g., “Love it, love it, love it, love it, love it.”

Care processes and outcomes

Users reported positive effects of eReferral on many processes and outcomes of care. Examples of the positive impacts are shown in Table 1. Many of these results are consistent with previous evaluations of eReferral.^{7, 14}

Table 1.

Overview of Positive Impacts of eReferral

Process/Outcome	Example Quotes
Clinical efficiencies	“…[The patient’s] visit is more productive because they’ve already done some of the initial diagnostic work.” (PCP) “It avoids patients that are completely inappropriately referred …which saves the patient and us time.” (specialty reviewer) “…so you can reassure the primary care physician, do this, this and that and if that’s all negative, [the patient] doesn’t need a referral.” (specialty reviewer)
Consult questions	“One of the big issues that used to come up–and it sounds absurd but–it would be very difficult to figure out what the primary provider wanted to have answered… So that’s not an issue anymore.” (specialty reviewer)
Wait times	“My sense…for GI, night and day. Liver clinic, night and day. It used to be five months to a year, now it’s about one to two months.” (PCP)
Administrative efficiencies	“Even if they’re not scheduled or were declined by Dr. [ ] …we could still see it on the history…it’s very accurate.” (specialty admin staff) “Less time is spent on calling, faxing.” (PCP admin staff)

Communication and coordination

The effect of eReferral on communication and coordination was central to improvements in care processes and outcomes. Many interview participants reported multiple positive effects of eReferral on communication, as illustrated in the following comment:

“It is a potentially very helpful and efficient system for referrals. …you can send information to the referral service and then get feedback in a timely manner that helps to triage the patient to them, also helps to educate the referrer…in terms of other work up that should be done beforehand, or other diagnoses that need to be considered.”

(PCP)

The system was perceived as helping co-management by providing support to the PCP for patients who would not see a specialist or in cases in which the reviewer determined that the patient did not need to seen in specialty care.

“That’s another advantage for some of these patients who just won’t come to our clinic; at least there’s a mechanism where you can provide some support for the primary care provider who’s stuck dealing with the problem.”

(specialty reviewer)

“So what happens is sometimes they get rejected or not scheduled, which is okay because sometimes it’s a question. It’s actually a communication, which I love because it goes directly to them. I get the answer in a couple of days.”

(PCP)

A related effect reported by many PCPs and specialists was the educational benefit of eReferral, chiefly for PCPs.

“I think most of the…eReferral people spend a fair bit of time explaining why we’re asking for tests or doing things, and through that I think it helps providers learn how to deal with some of these problems better on their own.”

(specialty reviewer)

“…and they can say, ‘Well, have you thought about this, have you thought about that? … And that is great for me because that expands my repertoire and my understanding of how to work up this kind of condition in the future.”

(PCP)

Other examples of benefits included ease and convenience of communication (no need to page or play voice-mail tag), opportunities to provide anticipatory guidance, and the impact of documentation on interactions which provided a record of ongoing discussions between providers, for example:

“Yes because we really didn’t have an interaction in the past. So this ability to interact, to send a note back to the referring physician who then sends you back a note, you know, there’s like a paper trail of what’s going backwards and forwards or computer trail let’s call it of what’s going backwards and forwards. So it makes much better interaction.”

(specialty reviewer).

Workflow and workload

From the perspective of PCPs, however, eReferral also had negative effects on work processes in that it shifted workload to them. This happened, in part, because some work that had been performed by administrative staff became the PCP’s responsibility, e.g.,

“It was a lot easier and quicker for me to write a consultation on…paper…Now I’m having to go through a longer process with a few more hurdles in it. Just mechanically if we have any problems with the computer. If General has any problems with the computer. If there’s a problem with a patient’s ID number. If the eReferral process suddenly disconnects, which happened quite a bit initially, and that has cleared up for the most part. …this was transferring work that was done by other people to the physician, and I wasn’t very happy about it”.

(PCP)

“…if somebody missed an appointment I could just go back and tell the scheduler reschedule it. Now…[the doctor] has to go back into his system and ask for it to be rescheduled.”

(PCP admin staff)

Increased workload for PCPs also occurred because eReferral transferred work from the specialist consultant. Prior to eReferral, when a patient arrived for his or her specialty appointment without prior work-up, the specialist would order the tests needed and schedule another visit. They also provided care for some problems that did not truly require specialty service. After eReferral, these elements of care that didn’t require the specialist were shifted to PCPs:

“I get a request for a particular test, sometimes tests that I, having seen and examined the patient, do not think is indicated. But [the patient] will not be scheduled [to see the specialist] unless they have gotten that test. …And then the onus will be on me to call the patient, and get the [test].”

(PCP)

“…the major downside is just that it creates a shared management system that pushes some of the responsibility for the consult on the referring physician… they’re in a very busy primary care practice, many…don’t have really great infrastructure.”

(specialty reviewer)

Despite these shifts, PCPs’ attitudes toward the system were generally positive due to effects of the system on quality of care and efficiency. For example, one PCP commented:

“It takes a little more effort to get on the computer, to work through the various screens. But I think it’s worth it… It used to just be every referral got at the end of the line. And now they’re actually reviewing cases individually and prioritizing them appropriately.”

On the receiving side, administrative staff reported less work due to eReferral. For example, the office manager for one of the specialty clinics reported that eReferral greatly reduced the need to handle scheduling problems, enabling greater attention to clinic operations. A scheduler reported that the time saved by eReferral enabled staff to work on a host of other administrative tasks.

eReferral also had a significant effect on workflow by changing the referral process to “batch” processing, which in turn resulted in unintended negative consequences on the ease of notifying patients about their appointments. In contrast to the prior paper-based system, where appointments in specialty care typically were scheduled while the patient was still in the primary care office, appointments made using eReferral were delayed until after specialist review and were scheduled with no knowledge of patient availability. The specialty clinic then sent the patient an appointment notification letter written in English plus a reminder letter two weeks before the visit. This process was especially problematic for patients who are homeless or do not have reliable contact information, who do not read English, or who may be more likely to ignore a letter than personal communication. These patients represent a substantial percentage of the population served. Both individual clinics and the development team implemented procedures to address the issues, described later in this article.

Technical Success Factors

Ease of use

Most users reported that the system was “easy to use” and “intuitive.” As a result, most users were satisfied with informal training, which consisted of a demonstration by a member of the development team or by a colleague. At the same time, users reported several specific issues with system design, network speed and access, and system integration, as summarized in Table 2. The consequences of most of these problems were extra time and effort for users, e.g., to restart a referral request or exit the system to obtain lab reports, although some issues led to supplementary processes or workarounds, described below. Design changes made to address specific issues are shown in [brackets].

Table 2.

Technical Issues with eReferral

Category	Issue
System design	Referring provider not automatically notified if a patient “no-shows” for their specialty appointment. Inability for someone who initiated communication to send another message until the recipient replied to the previous message. [Message threading redesigned to enable either participant to add an additional message whenever needed.] Notification of an appointment sent only to the referring provider, not other primary care staff. [Notification of additional staff enabled for tracking.] Inability to save a draft of the electronic referral, resulting in the need to re-input the information if the system timed out or lost connectivity or if the user exited the system before submitting the referral request. [“Save draft” feature added.] Need to submit a new referral request rather than re-submit an existing request if the patient missed an appointment. [Modified program to enable resubmission of a scheduled referral request up to 6 months after the appointment date.] Inability to specify patient scheduling availability for a specialty appointment. [Added a field for notes to scheduler on patient’s expected or known constraints.]
Network speed and access	Need to log in to multiple systems/levels. Inability to access system remotely. Slow connections. [Network and computers were upgraded in several referring sites.]
System integration	Need for specialist reviewers to exit eReferral to view imaging studies or lab reports in the EHR. [Feature added to append available test results to the eReferral.] Lack of notification when test results were available, which were needed for the PCP to submit the eReferral request. [Feature added to allow PCPs to flag an eReferral for a designated time period (e.g., 2 weeks, 4 weeks), after which time an automated reminder email is sent to the PCP to check his/her worklist.] Lack of integration with specialty clinic’s scheduling system.

Supplementary processes and workarounds

Several clinics developed new processes to address some of the issues shown in Table 2 or to otherwise improve eReferral functionality. Staff in 3 primary care clinics created logs or spreadsheets to track the status of referrals (e.g., date of appointment, patient notification), entering data manually or cutting and pasting fields from eReferral screens. Some primary care clinics produced and sent appointment notification and reminder letters in the primary languages of their patient populations (e.g., Chinese, Korean, and Russian). In addition to new work processes, some respondents reported using workarounds. The most frequently reported workaround was “fudging” information in order to get around some scheduling requirements posted on the specialty clinic’s policy page. For example, some PCPs would “check the box” e.g., report that tests were pending when the provider felt that the requirement was unnecessary or would substantially delay patient care. Another workaround involved going outside of eReferral (calling or faxing) to request a referral or modify an existing request. In fact, some PCPs would call the specialty clinic at specific times of day because they knew that particular staff would accept changes over the phone. Yet another workaround in primary care clinics involved instructing hard-to-reach patients to return to the clinic in a week or two to find out their specialty care appointment time.

Implementation Practices

Implementation pace and system support

The measured pace of the implementation, coupled with a responsive development team, were important factors contributing to adoption. eReferral was implemented as a pilot in one specialty clinic for the first 18 months before it was launched in a second clinic. This demonstrated early benefits of the system for many subsequent users, who then had positive performance expectancies¹⁶ for the system, as noted by an administrative staff member in primary care, “I think the reason why they’ve jumped onto the bandwagon is because they probably saw how efficient it was with GI.” Moreover, the pilot, as well as a phased rollout, enabled the development team to make continual improvements to the system. The team resolved a number of the design issues, such as providing a field to specify dates or times when the patient is unavailable for an appointment, developing worklists for non-clinical staff to track providers’ referral requests, allowing the PCP to save a draft of the referral request and to re-submit a request for “no-shows”, and adding an indicator to providers’ work lists indicating when there has been activity on a referral request. The team also facilitated changes in organizational practices; notably, they brokered revision of some specialty practices’ policies for referral requests that many PCPs viewed as unreasonable, such as a requirement in orthopedics that users check a box indicating that the patient was willing to have surgery.

Social influence

Influence from important others is a construct in a number of models of technology adoption.^{15, 16, 22} Many interviewees identified clinicians from the eReferral development team as champions for the system. However, interviewees in two clinics identified their medical director as a champion for eReferral; for example, one medical director solicited input from staff about the system at weekly clinic meetings, trained the medical assistants, wrote a policy manual for eReferral use, and developed procedures, including the tracking spreadsheet and notification letters in multiple languages described earlier. Many interviewees reported strong peer support, although this varied somewhat, particularly among PCPs who objected to increased workload brought about by eReferral.

Other Facilitating Conditions

Interview participants identified other conditions that affected implementation.¹⁶ As noted earlier, most users were satisfied with initial levels of training provided. However, interviews conducted after an additional year of use revealed that many users were not aware of new features in eReferral (e.g., those denoted in Table 2), in spite of the implementation team’s prolific efforts to communicate design changes. Thus, although most users seemed to be adept at using the basic features of eReferral, they may not have been exploiting the full functionality of the system.

Specialty reviewers as well as some PCPs reported having sufficient access to hardware, and these users generally had no complaints about connection speed. Some primary care clinics, however, experienced inefficiencies due to a shortage of computers and lack of computer access in exam rooms. Providers reported having to shuttle between exam rooms and the computer to get information from the patient for the eReferral or inputting the eReferral after the visit, which sometimes necessitated calling the patient for information. These users also reported problems with the connection “freezing.” Nonetheless, some of the users in these clinics were the most ardent supporters of eReferral because of the impact of the system on access to specialty care and perceptions of improved patient health and safety outcomes.

Discussion

This case study provides rich documentation of the outcomes from a successful electronic referral program, and it suggests a range of design features and implementation factors that accounted for the program’s success. Users perceived that eReferral largely prevented the occurrence of low-value specialty visits due to unclear consult questions, incomplete workups, and referrals for problems that could be managed in primary care. The system was also perceived as having markedly reduced wait times for specialty services, which had previously been up to a year for some specialties in this historically under-resourced setting.

These successes were achieved because PCPs and specialty reviewers were willing to spend time using the system rather than circumventing it. For most reviewers, this effort was part of their compensated clinical time. However, PCPs were not compensated for the additional time required to submit and manage referrals, order preliminary tests, and manage or co-manage patients. Some of these tasks were particularly burdensome in clinics with poor technical infrastructure. Likewise, increased workload or time has been reported in other studies of HIT.^{23, 24} Nonetheless, PCPs (and other users) were generally enthusiastic about eReferral, chiefly due to the professional satisfaction of gaining improved access to specialty care for their patients. PCPs also valued the unique opportunities to learn and to gain reassurance from the dialogue with specialists that eReferral enabled. This result was a direct product of system features that enabled asynchronous communication between the parties to the referral.

Although the primary factor motivating the use of eReferral was its perceived benefits, several other factors played a role in the system’s success, including an intuitive user interface that minimized training needs, the development team’s readiness to add or modify features in response to users’ needs, a measured pace of rollout, strong physician champions, and the distinctive process adaptations used in some clinics (such as use of follow-up tracking sheets). The mandatory nature of the system also undoubtedly promoted adoption. However, even in large-scale technology adoption efforts, users will find ways to get around the system if it is difficult to use or has negative consequences.²⁵ This was rarely observed for eReferral.

The system appeared to have some unintended workflow consequences. One was a shift in administrative tasks to PCPs. While the transfer of tasks such as test-ordering and patient management from specialist to PCP is arguably appropriate, the transfer of administrative tasks to PCPs is inefficient. A second problem was the inability of patients to participate in setting appointment times for their specialty care visits, a particular problem for the large percentage of homeless and limited English-speaking patients in the clinics’ populations. As in other studies,²⁵ users responded to these effects of the system on workflow by adopting supplementary processes and workarounds. These workflow issues were addressed in eReferral through system design changes. Several studies have emphasized the need for analysis of workflow and redesign prior to implementing HIT.^26–28 These results also point to the need to continue seeking and adapting to unintended consequences post-implementation.

Overall, the success of eReferral may be most attributable to the “human-centered” approach²⁹ taken in its development. The design team considered workflow at the system development stage. They used participatory design strategies, with clinicians who understood user needs serving as key members of the development and implementation teams. The in-house software team, in combination with ongoing evaluation, enabled continual improvement through system design changes rather than simply through training and workflow adaptations. Coupled with a measured pace of roll-out and strong social influence, the design process may have enabled a virtuous cycle in which each specialty clinic added to eReferral led to design changes that improved usability, thereby enhancing user perceptions and increasing the demand for more specialty clinics to be brought online. This example stands in contrast to current EHR implementation plans in many healthcare organizations, where new systems are being fielded rapidly with limited opportunities to make design changes. Our findings are also consistent with result from a systematic review that found better outcomes from “home-grown” e-prescribing systems.³⁰

An ongoing challenge in implementation of eReferral is ensuring that users become familiar with system changes. Consistent with other studies,^{31, 32} we found that at least some users were unaware of system features, particularly for changes that had been made to correct initial problems. This occurred despite announcements and demonstrations of these features in outreach meetings, newsletters, and email broadcasts. Bundling changes and rolling them out on a less frequent or a more predictable schedule may be one strategy for helping users stay up-to-date. Another, more focused strategy would be to use system logs to identify users who are not taking advantage of new features in order to target training toward those individuals.

An important limitation of the study is its uncertain generalizability beyond the safety net delivery system in which specialists are salaried and PCPs face sometimes daunting challenges in obtaining specialty access. However, integrated delivery systems such as staff-model health maintenance organizations and possibly newer arrangements such as accountable care organizations have the incentives to compensate specialists and PCPs for the shifts in effort that eReferral would require. In addition, improved communication and coordination with specialists would likely be welcomed enthusiastically by PCPs in most communities. The eReferral model may also be a useful tool in the Patient Centered Medical Home, provided that the incremental PCP time is adequately covered by the coordination fees included in that model. Our study was also limited to interviews and did not employ other usability methods, such as user surveys and observations.²⁹ However, separate surveys of PCPs⁷ and specialists¹³ found consistent results. Another possible limitation was that if medical directors selected users who were favorably disposed toward eReferral to participate in the interviews, we may have missed reports of downsides of the system or implementation challenges.

In fee-for-service settings, an eReferral model might be acceptable to specialists if insurers were to compensate specialist reviewers for their time or if specialists were able to shift their efforts toward higher-value services. In areas of specialist oversupply, the latter strategy may not work because of the difficulty of filling in the lost low-value services through gains in market share. Future work would be needed to address each of these challenges. Knowledge-based methods for collecting more structured data and for intelligent protocols could partially automate the review process, reducing the expense of the specialist review. At the same time, it would be critical to monitor and enhance the satisfaction of PCPs and patients with the referral process, using technology to enhance rather than to degrade the sense of connection to the specialist.