Follow-up Actions on Electronic Referral Communicationin a Multispecialty Outpatient Setting

ABSTRACT

OBJECTIVES

Electronic health records (EHR) enable transmission and tracking of referrals between primary-care practitioners (PCPs) and subspecialists. We used an EHR to examine follow-up actions on electronic referral communication in a large multispecialty VA facility.

METHODS

We retrieved outpatient referrals to five subspecialties between October 2006 and December 2007, and queried the EHR to determine their status: completed, discontinued (returned to PCP), or unresolved (no action taken by subspecialist). All unresolved referrals, and random samples of discontinued and completed referrals were reviewed to determine whether subspecialists took follow-up actions (i.e., schedule appointments anytime in the future) within 30 days of referral-receipt. For referrals without timely follow-up, we determined whether inaction was supported by any predetermined justifiable reasons or associated with certain referral characteristics. We also reviewed if PCPs took the required action on returned information.

RESULTS

Of 61,931 referrals, 22,535 were discontinued (36.4%), and 474 were unresolved (0.8%). We selected 412 discontinued referrals randomly for review. Of these, 52% lacked follow-up actions within 30 days. Appropriate justifications for inaction were documented in 69.8% (150/215) of those without action and included lack of prerequisite testing by the PCP and subspecialist opinion that no intervention was required despite referral. We estimated that at 30 days, 6.3% of all referrals were associated with an unexplained lack of follow-up actions by subspecialists. Conversely, 7.4% of discontinued referrals returned to PCPs were associated with an unexplained lack of follow-up.

CONCLUSIONS

Although the EHR facilitates transmission of valuable information at the PCP-subspecialist interface, unexplained communication breakdowns in the referral process persist in a subset of cases.

BACKGROUND

Referrals in the outpatient setting are an essential component of primary care^1,2. Previous literature has addressed the importance of good communication between primary care practitioners (PCPs) and subspecialists^3–8, and has highlighted basic principles of PCP-subspecialist relationships^9–13. However, referral communication is susceptible to breakdowns^3,14–16. In one study, approximately 68% of subspecialists received no information from referring practitioners prior to seeing the patient, and information from subspecialists was transmitted back to PCPs only 55% to 80% of the time¹⁴. Paper-based referrals are especially hard to track and are vulnerable to loss in the fragmented outpatient environment where information transfer occurs between practices and often between physical locations¹⁷. Electronically transmitted referrals may overcome many of the risks inherent to the physical routing and handling of paper-based requests¹⁸. Furthermore, referrals generated using integrated electronic health records (EHRs) allow practitioners to easily document the reasons for referral, facilitate subspecialist access to relevant clinical information, and allow subspecialists to readily communicate questions or findings back to the referring practitioner¹⁹.

To our knowledge, few published studies have focused on PCP-subspecialist communication through integrated EHR systems, and none have assessed the timeliness of follow-up actions taken by recipients. However, this topic is important for two reasons. First, in some systems, resources for subspecialists are constrained. This is particularly true of systems that serve uninsured or underinsured populations, for which disparities in care may already exist²⁰. Subspecialists in such systems have fewer financial incentives to accept and act on marginally appropriate referrals; thus, they may “triage” or prioritize referrals and discontinue referrals in certain cases (e.g., returning the referral to PCP when clinical information in the referral is inadequate). Second, despite advantages of electronic transmission, other types of EHR-based communication (e.g., test result notification) were recently found to be susceptible to breakdown^21,22. The aim of the present study was to assess timely follow-up of referral communication through an integrated EHR. We focused on the two-way communication at the PCP-subspecialist interface and identified referrals without timely follow-up actions and potential reasons for inaction in these cases.

METHODS

Design

We conducted a retrospective chart review to evaluate follow-up actions taken by subspecialists within 30 days of receiving an electronic referral at a Veterans Affairs (VA) facility that uses an advanced integrated EHR. We defined timely follow-up as scheduling, within 30 days of referral receipt, a subspecialty clinic appointment for any time in the future. For cases without appropriate follow-up action at 30 days, we further reviewed the EHR to determine if there was a justifiable reason to support the lack of action. We also reviewed the EHR to determine PCPs’ follow-up actions when their referrals were discontinued or returned by the subspecialist.

Setting

We studied referrals within the multispecialty outpatient setting of a tertiary care VA facility and its five satellite clinics. The local institutional review board approved the study. VA practitioners request referrals through an order entry system in the EHR. Per institutional policy, subspecialty services must respond to all requests for referrals within 7 days. Subspecialists may either schedule an appointment, based on the availability and urgency of the case, or discontinue the request if it is deemed inappropriate or incomplete. In general, subspecialty resources in the VA are capped, and there are no additional financial incentives for subspecialists to see additional patients or perform additional procedures. The number of subspecialists varies between facilities, and also depends on facility size and population served. For instance, to support approximately 120,000 unique patients at the study facility, currently there are 14 cardiologists and 9 gastroenterologists.

Referral Sampling Frame

We retrieved all outpatient referrals made at the study site between October 15, 2006 and December 15, 2007 to five subspecialty services: cardiology, gastroenterology, neurology, pulmonary, and surgery. Subspecialties were chosen based on their high volume of requests and because referrals to these subspecialties tend to be relatively time-sensitive. Forty-five days after the last referral request in our study population, we queried the EHR to determine the electronically documented status of each referral as follows:

1. Completed: The subspecialty service resolved the request (e.g., the patient was evaluated).

2. Discontinued: The subspecialty service did not accept the referral for a specified reason or considered the reason for referral to be inappropriate.

3. Unresolved: At the time of data extraction, there was no documented response to the referral.

Chart Review Procedures

A standardized data collection instrument was designed and pilot tested by the study team. To ensure reliable data collection, reviewers received hands-on training from the study team on use of the data collection instrument. After initial training, each reviewer rated 20 records independently. The study team then reviewed the data and checked it against the medical record for inconsistencies. This process was repeated to ensure reliable and consistent data collection for each reviewer before they were allowed to collect data independently. After training, each chart was evaluated by two physician reviewers (either chief resident/fellow or staff) and disagreements related to three variables (potential for harm, appropriateness of provisional diagnosis, and appropriateness of urgency) in the referral were resolved by consensus between the reviewers.

Reviewers were not limited to the information contained in the referral and informed their decisions by reviewing progress notes, test results, and other pertinent information in the EHR. The reviewers documented the type of ordering practitioner, provisional diagnosis for referral, and any notation of urgency, if indicated. Reviewers also judged whether the assigned priority flag (such as marking of urgency <7 days, <30 days, next available, or routine) was appropriate and whether the PCP’s provisional diagnosis was appropriate (e.g., a request for a colonoscopy on a patient with a provisional diagnosis of suspected colon cancer would be considered appropriate, but a similar request with a provisional diagnosis of diabetes would not). Reviewers then determined whether appropriate follow-up action (i.e., scheduling future appointments) occurred within 30 days after each referral (a timeline we used in previous work on electronic communication of laboratory results)²¹, regardless of documented status retrieved from the EHR. For instance, a patient with an unresolved referral may have seen the subspecialist during an inpatient visit or may have secured an appointment through means other than the referral process.

For referrals with no evidence of follow-up at 30 days by the subspecialist, we determined whether or not the inaction was supported by a predetermined, justifiable reason. For referrals that were discontinued by the subspecialty service and returned to the PCP, reviewers determined whether the PCP needed to take subsequent action and if that action was taken. Finally, reviewers assessed whether unexplained inactions were associated with any potential for patient harm. Potential for harm was scored from 0 (referral completed, no potential for harm) to 6 (virtually certain evidence that patient harm could result) using an instrument adapted from a previous study (Table 1)²³. All ratings were assigned based on reviewers’ clinical judgment after extensive review of records and after considering the patient’s current medical status, past medical history, family history, symptoms, and potential clinical outcomes if the condition was left undiagnosed or untreated.

Table 1.

Potential for Harm Rating Scale (Adapted from Gandhi et al. 2006, Ref. 23)

0	Not applicable—referral completed
1	Little or no evidence that patient harm could result
2	Slight to modest evidence that patient harm could result
3	Not quite likely that patient harm could result; less than 50-50, but close call
4	More likely than not that patient harm could result; more than 50-50, but close call
5	Moderate/strong evidence that patient harm could result
6	Virtually certain evidence that patient harm could result

Sample Size Calculation

We calculated sample size based on exploratory preliminary study aims that focused on why referrals may have been unresolved. On an initial review of 100 unresolved referrals, we found that a provisional diagnosis was appropriate in only 83%. We thus estimated that an appropriate provisional diagnosis would be expected in a higher number of completed referrals and a lower number of discontinued referrals. Assuming that 90% of completed referrals and 50% of discontinued referrals would have appropriate provisional diagnoses, a sample size of at least 139 completed and 385 discontinued requests would be required (95% confidence interval with 5% margin of error). We chose our final sample size taking into account this calculation and the total number of referrals for each subspecialty.

Data Analysis

We compared the distributions (as proportions) of the following independent variables among the three groups (completed, discontinued, and unresolved): referring practitioner type, urgency of referral (routine, urgent, unspecified), and the appropriateness of urgency and provisional diagnosis. In the discontinued and unresolved groups, we categorized referrals lacking follow-up action at 30 days into subgroups with and without a justifiable reason, and for the latter subgroup we calculated mean potential-for-harm scores. Finally, we tabulated descriptive data for the justifiable reasons for lack of follow-up actions at 30 days. We used chi-square tests to compare the distributions of categorical variables and Fisher’s exact test when the assumptions for chi-square were not met. We used SAS (SAS Institute Inc., Cary, NC, version 9.1) to conduct all statistical analyses.

RESULTS

We retrieved 61,931 outpatient referrals transmitted via the EHR to the five subspecialty services during the study period. Of these, 38,922 referrals were completed (62.8%), 22,535 were discontinued (36.4%), and 474 were unresolved (0.8%); see Figure 1. We randomly extracted 159 and 412 referrals from the completed and discontinued groups, respectively, and we reviewed all 474 unresolved referrals.

Figure 1.

Study flowchart.

On chart review, we found no evidence that action was taken within 30 days of referral request in 17.6% (28/159) of completed referrals (i.e., appointment was not given within 30 days even though the patient might have been seen at a future date), 52.2% (215/412) of discontinued referrals, and 84.4% (400/474) of unresolved referrals. There was no justified documented reason for inaction in the majority (77%) of unresolved referrals (Fig. 1). Unexplained or unjustified inaction was found in 15.8% of the 412 discontinued referrals. Extrapolating this rate to the 22,535 discontinued referrals and accounting for 474 unresolved referrals, we estimated that 6.3% of 61,931 referrals did not lead to timely follow-up actions by subspecialists. Table 2 shows the distributions of the independent variables across the three groups. The provisional diagnosis was appropriate in the majority of referrals (95% of completed, 87.8% of unresolved, and 91.7% of discontinued referrals). Similarly, the urgency flag was also appropriate in the majority (96.2% completed, 85% unresolved, and 92% discontinued referrals).

Table 2.

Characteristics of Completed, Unresolved, or Discontinued Referrals

Characteristics	Completed (n = 159)	Unresolved (n = 474)	Discontinued (n = 412)
	n (%)	n (%)	n (%)
Referral type
Cardiology	43 (27.0)	94 (19.8)	70 (17.0)
Pulmonary	15 (9.4)	12 (2.5)	15 (3.6)
Gastroenterology	15 (9.4)	116 (24.5)	112 (27.2)
Neurology	16 (10.1)	119 (25.1)	42 (10.2)
Surgery	70 (44.0)	133 (28.1)	173 (42.0)
Ordering practitioner
Staff physician	92 (57.9)	233 (49.2)	221 (53.6)
Nurse practitioner	16 (10.1)	53 (11.2)	33 (8.0)
Physician assistant	31 (19.5)	75 (15.8)	98 (23.8)
Trainee	20 (12.6)	113 (23.8)	58 (14.1)
Other	0 (0.0)	0 (0.0)	2 (0.5)
Referral status
Routine	122 (76.7)	406 (85.7)	324 (78.6)
Urgent	4 (2.5)	4 (0.8)	2 (0.5)
Unspecified	33 (20.8)	64 (13.5)	86 (20.9)
Provisional diagnosis
Appropriate	151 (95.0)	416 (87.8)	378 (91.7)
Inappropriate	8 (5.0)	58 (12.2)	34 (8.3)
Urgency flag
Appropriate	153 (96.2)	403 (85.0)	379 (92.0)
Inappropriate	6 (3.8)	71 (15.0)	33 (8.0)

Table 3 lists the mean potential for harm ratings in discontinued and unresolved referrals that lacked follow-up action at 30 days and had no documented justification for inaction. The mean potential harm rating scores in the two groups were 1.7 (SD = 1.3) and 2.4 (SD = 1.1), respectively. For unresolved referrals, the mean potential harm scores differed significantly across subspecialties and also as a function of referral status and the appropriateness of the provisional diagnosis and urgency notation. For discontinued referrals, however, the mean potential for harm score was not associated with any of the referral characteristics.

Table 3.

Potential for Harm Associated with Referrals that Were Unresolved and Discontinued without Documented Justification of Absence of Appointment at 30 days

Characteristics	Referrals without documented justification
	Unresolved (n = 365)		Discontinued (n = 65)
	n (%)	Mean potential for harm (SD)	n (%)	Mean potential for harm (SD)
Referral type
Cardiology	60 (16.4)	1.3 (0.7)	6 (9.2)	2.3 (2.0)
Pulmonary	2 (0.5)	3.5 (0.7)	1 (1.5)	1.0 (0.0)
Gastroenterology	95 (26.0)	2.5 (0.8)	21 (32.3)	1.8 (1.4)
Neurology	109 (29.9)	2.3 (0.8)	7 (10.8)	1.4 (0.8)
Surgery	99 (27.1)	3.2 (1.2)	30 (46.2)	1.6 (1.3)
Ordering practitioner
Staff physician	190 (52.1)	2.4 (1.2)	32 (49.2)	1.7 (1.4)
NP	61 (16.7)	2.5 (1.0)	18 (27.7)	1.4 (0.9)
PA	26 (7.1)	2.4 (1.2)	5 (7.7)	2.0 (2.2)
Trainee	88 (24.1)	2.4 (1.0)	9 (13.8)	2.2 (1.6)
Other	–	–	1 (1.5)	1.0 (0.0)
Referral status
Routine	314 (86.0)	2.3 (1.1)	51 (78.5)	1.7 (1.3)
Urgent	1 (0.3)	1.0 (0.0)	–	–
Unspecified	50 (13.7)	3.0 (1.3)	14 (21.5)	1.6 (1.4)
Provisional diagnosis
Appropriate	329 (90.1)	2.5 (1.1)	62 (95.4)	1.7 (1.3)
Inappropriate	36 (9.9)	2.0 (1.1)	3 (4.6)	2.0 (1.7)
Urgency flag
Appropriate	320 (87.7)	2.3 (1.0)	57 (87.7)	1.6 (1.2)
Inappropriate	45 (12.3)	3.3 (1.3)	8 (12.3)	2.3 (2.3)

Table 4 lists reasons to explain why there was no follow-up action at 30 days for a certain proportion of discontinued and unresolved referrals. Common reasons included no prerequisite testing by the PCP, the subspecialist’s opinion that no intervention was required, and requests to the wrong service. In 31 (7.4%) of the discontinued referrals (with or without a justified reason), there was no evidence of further follow-up or justification for inaction by the PCP. The mean potential for harm rating in this subgroup of referrals was 2.8 (SD = 1.6).

Table 4.

Reasons Why Absence of Appointment at 30 days Was Justifiable

Reason	Discontinued (n = 150)	Unresolved (n = 35)
	n (%)	n (%)
Patient was subsequently admitted and outpatient referral not required	–	4 (11.4)
Duplicate referral	17 (11.3)	6 (17.1)
Patient left the institution	–	1 (2.9)
Patient received care elsewhere	4 (2.7)	1 (2.9)
Patient refused	4 (2.7)	–
No intervention required by consultant	29 (19.3)	1 (2.9)
Prerequisite testing not performed by PCP	30 (20.0)	14 (40.0)
Not enough information given on referral	14 (9.3)	–
Too many patient comorbidities that precluded a procedure	7 (4.7)	–
Wrong service consulted	27 (18.0)	–
Wrong study requested	3 (2.0)	–
Referral later discontinued by ordering provider	9 (6.0)	2 (5.7)
Subspecialists called in wait-listed patients for appointments soon after 30 days	5 (3.3)	2 (5.7)
Other	1 (0.7)	4 (11.4)

DISCUSSION

Electronically transmitted referrals have the potential to improve patient follow-up because they ensure delivery of information and can be tracked, in contrast with paper-based referrals. We found that over a third of referrals were initially discontinued and returned to the PCP by the subspecialists; however, timely follow-up actions (scheduling future subspecialty appointments) still resulted in about half of these cases. In the majority of cases without timely follow-up, subspecialists most often cited lack of prerequisite testing by the PCP and no necessity for subspecialist intervention as reasons for discontinuation of referrals. However, in over 15% of discontinued referrals, we did not find any explanation for the lack of follow-up by the subspecialist within 30 days of referral transmission. Extrapolating from our sample to the entire population of the referrals we extracted, we estimated that this scenario occurred in about 6% of all referrals. Meanwhile, approximately 7% of discontinued referrals received no further follow-up from PCPs. Our study thus reveals that breakdowns in referral communication might occur even when referrals are transmitted through an integrated EHR.

Although 474 referral requests in our study appeared to have no documented response from a subspecialist, these represented a relatively small proportion of referrals overall. Of greater concern is the relatively large percentage of discontinued referral requests that required additional follow-up action from the PCP where no action was documented. Documentation and execution of necessary next steps in the referral process represent vulnerabilities for timely patient follow-up and safe patient care. Our findings suggest there is ample room for improving the electronic referral communication process, which may in turn reduce delays in care and potential harm. To our knowledge, this is the first study to evaluate timeliness of follow-up of referral communication through an integrated EHR.

Improving the outpatient referral process poses several challenges. In discontinuing referrals subspecialists stated justifications, such as lack of prerequisite workup, which suggest possible disagreement on criteria for referral to their services. This is consistent with a previous study of communication breakdowns in which subspecialists and PCPs did not consistently agree about the reasons for referrals²⁴. In a few cases, subspecialists did not believe sufficient information was provided in the referral to make an informed decision. Both quality and quantity of information provided in referrals need further study; a recent study found that more than one in four referrals did not contain a clear clinical question²⁵. Future work may need to focus on developing, adopting, and integrating referral criteria specific to the services receiving referrals and testing EHR-based referral templates that reflect such criteria.

It was also concerning that 7.4% of discontinued referrals that required follow-up actions by the PCPs did not always lead to timely response. Given the large volume of discontinued referrals, we believe this to be of great significance. In the VA system, practitioners have the option of turning off certain referral-related notifications, including those alerting them about referral discontinuation. Although we were unable to track whether the practitioners involved had indeed turned off referral notifications, our previous work has shown that certain high-priority notifications might still be overlooked within the EHR despite the notification receipt^21,22. Thus, our work establishes the need for reliable systems-based tracking procedures to identify communication breakdowns related to electronic referrals.

Based on our findings we propose several potential strategies to improve the referral communication process and lay the foundation for future research. First, in order to develop system-based interventions in this area, a better understanding of variation in referral communication processes and policies across settings and subspecialty services is needed. For instance, responsibilities of PCPs and subspecialists should be better defined in the context of referrals and key processes and best practices should be shared and standardized across subspecialty services, when possible. Potential interventions to reduce discontinuations include clear referral criteria, developed with input from both PCPs and subspecialists, clarifying which patients are “eligible” for referrals. Second, effective procedures to ensure appropriate follow-up of unresolved or discontinued referrals in institutions using integrated EHRs must be established. For instance, implementation of a system to track referrals could perhaps prompt a non-provider when referral information has not been acted upon in a defined time period. Third, ensuring that PCPs provide adequate information to the subspecialist, perhaps through EHR-based referral templates^17,26, will minimize the risk for discontinuation of the request and reduce valuable subspecialist time spent on redundant chart reviews. Although this may not be an easy goal to achieve, automated information extraction tools that populate the electronic referral template could ease the burden of completing a detailed referral template²⁶. Finally, PCPs must state a clear referral question to avoid ambiguity about the need for referral, and EHR systems should provide a distinct mechanism for this, perhaps including a list of generic referral questions (e.g., “Does this patient need X procedure?” or “Can you help adjust X medication?”).

Our study has several limitations. We sampled from a single institution and focused on outpatient referrals to a specific set of subspecialties. Our findings thus may not easily generalize to other sites or subspecialties or to the inpatient setting. Nevertheless, they provide significant safety insights for health care systems where subspecialist access may be capped and provide valuable lessons on EHR-based referral communication that may be applicable to other settings. We also did not include any direct comparative evaluation with a paper-based system. While such a comparison would have certainly provided valuable findings, communication processes in a sample size of comparable magnitude would be much harder to track in paper-based referrals. Additionally, communication breakdowns related to referrals have already been described in paper-based systems^3–5. Although we relied on medical record review data for certain implicit clinical judgments, we used consensus to reduce bias. We also used an arbitrary time period of 30 days to determine follow-up actions by subspecialists. However, currently there are no accepted standards on timeliness or what constitutes a delay in referral communication except those suggested by the VA, consequently our work could lead to future knowledge generation in this area. Lastly, we did not measure actual harm from delays in referral care. The mean potential harm scores in our study were overall low, suggesting that harm was unlikely or minimal from many of these delays, even though some patients may have been at risk for significant harm.

In conclusion, we found that an EHR facilitated a valuable interchange of information at the PCP-subspecialist interface; however, unexplained lack of follow-up on communicated information by both PCPs and subspecialists persisted in a subset of cases. To improve the timeliness and efficiency of the referral process, future research and interventions should aim to standardize referral communication processes, design effective referral tracking mechanisms, and develop and adopt strategies to reduce the potential for communication breakdowns.