Impact of Implementation of Electronically Transmitted Referrals on Pediatric Subspecialty Visit Attendance

Abstract

Objective

One barrier to timely access to outpatient pediatric subspecialty care is the complexity of scheduling processes. We evaluated the impact of implementing electronically transmitted referrals on subspecialty visit attendance.

Methods

Through collaboration with stakeholders, an electronically transmitted referral order system was designed, piloted, and implemented in 15 general pediatrics practices, with 24 additional practices serving as controls. We used statistical process control methods and difference-in-differences analysis to examine visits attended, appointments scheduled, appointment nonattendance, and referral volume. Electronically transmitted referrals then were expanded to all 39 practices. We surveyed referring pediatricians at all practices before and after implementation.

Results

From April 2015 through September 2016 there were 33,485 referral orders across all practices (7770 before the pilot, 11,776 during the pilot, 13,939 after full implementation). At pilot practices, there was a significant and sustained improvement in subspecialty visits attended within 4 weeks of referral (10.9% to 20.0%; P < .001). Relative to control practices, pilot practices experienced an 8.6% improvement (P = .001). After implementation at control practices, rates of visits attended also improved but to a smaller degree: 11.8% to 14.7% (P < .001). In survey responses, referring pediatricians noted improved scheduling processes but had continued concerns with appointment availability and referral tracking.

Conclusions

While electronically transmitted referrals improved visit attendance after pediatric subspecialty referral, the sizable percentage of children without attended visits, the muted effect at control practices, and pediatrician survey responses indicate that additional work is needed to address barriers to pediatric subspecialty care.

Nearly a quarter of families of children in need of subspecialty care report difficulty accessing that care.¹ Multiple barriers contribute to this difficulty, including limited subspecialist availability, excessively lengthy travel times, and costs.^2–6 One additional barrier is the complexity of scheduling processes.^7–9 Families report uncertainty about scheduling processes and excessive time spent attempting to make appointments,⁸ while primary care physicians (PCPs) and subspecialists report inadequate transfer of information between physicians and difficulty tracking referrals.^10,11

One proposed method of improving scheduling issues is electronically transmitted referral order systems in which PCPs electronically transmit referral orders to subspecialists.¹² Implementation of electronically transmitted referrals has been associated with improved satisfaction among PCPs of adult patients.¹² Of note, electronically transmitted referrals are distinct from electronic consultations (e-consults), which facilitate iterative communication between PCP and specialists regarding clinical questions (or electronic “curbsides”) through a store-and-forward platform.^13,14 e-Consult systems require significant changes in referral processes, information transfer, and clinician expectations. In contrast, electronically transmitted referrals involve only the information traditionally provided via paper referrals but transmit this information electronically to subspecialist office staff. Electronically transmitted referrals are used in multiple health systems, but the impact of implementation of electronically transmitted referrals has not been described in pediatric settings.

At our institution, timely access to pediatric subspecialty care was identified as a target for improvement. A collaborative effort identified electronically transmitted referrals as a potential mechanism for improving referral processes and patient access. In this evaluation, we aimed to monitor the impact of implementing electronically transmitted referrals by assessing referral processes and outcomes before and after implementation. We identified subspecialty visits attended within 4 weeks of referral as the primary outcome for evaluating improvement. We hypothesized that improving scheduling processes would improve appointments scheduled and visits attended, but could also increase appointment nonattendance.

Methods

Context

The Children’s Hospital of Pittsburgh (CHP) of University of Pittsburgh Medical Center (UPMC) is a freestanding academic 315-bed children’s hospital in southwestern Pennsylvania. Over 240,000 outpatient subspecialty visits occurred in 2015 at the main hospital and 9 satellite sites. Thirty-nine community general pediatric practices affiliated with CHP participated, each with a shared electronic medical record (EMR) and with locations ranging from 1.5 to 128 miles from CHP. Patients from these practices account for 28% of CHP subspecialty visits. At baseline, PCPs generally referred patients by providing telephone numbers to the family (ie, without direct communication with subspecialist offices). At baseline, PCPs could enter referral orders in the EMR (“EMR referral orders”) for insurance and documentation purposes, but EMR referral orders were not electronically transmitted and were not required for all referrals. At baseline, among parents of children seen by CHP subspecialists, only 52.3% reported maximum satisfaction with access domains on the Consumer Assessment of Healthcare Providers and Systems (CAHPS) survey, prompting interest in improving access to subspecialty care.

Intervention

Stakeholders including general pediatricians, subspecialists, and hospital leadership began informally meeting in February 2015 to discuss improving subspecialty referrals processes prompted by CAHPS survey results. Several opportunities for improving referral processes were identified, broadly divided into issues of appointment scheduling, appointment availability, and visit attendance (Fig. 1A). The group prioritized appointment scheduling because these were the first barriers patients encountered. A 5-member core group (including representatives of general pediatricians, subspecialists, operational leadership, informaticians, and implementation specialists) met weekly, then biweekly, and then monthly from June 2015 through May 2016 to design, pilot, and implement an intervention to improve the scheduling process using electronically transmitted referrals. The core group also met regularly with additional broader groups, including hospital leadership, subspecialist practice managers, subspecialist schedulers, primary care executive leadership, primary care physicians, and primary care practice managers, to review plans, provide education and support, and obtain feedback.

Figure 1.

Subspecialty referral process maps. Process maps reflecting steps between referral, appointment scheduling, and appointment attendance both (A) before electronically transmitted referrals and (B) with electronically transmitted referrals. “Family-centered” appointment here indicates the availability of appointments that are feasible for families to attend in the context of other obligations.

The electronically transmitted referral intervention included 3 main changes (Fig. 1B). First, the EMR referral order was redesigned to transmit electronically to CHP sub-specialty schedulers through a shared electronic health platform for referrals specific to CHP specialists. To facilitate this, a new field was added for PCPs to indicate desired site of subspecialty care (“CHP” or “other”). When “other” was selected, referral orders functioned as preexisting EMR referral orders (ie, they were not electronically transmitted). Second, for referrals electronically transmitted to CHP, schedulers then called families up to 3 times. As shown in the second process map, these two steps bypassed many steps and decisions that the family would otherwise need to navigate (ie, understanding the need for referral, deciding to schedule the referral, actually calling the scheduler, and navigating the phone tree). Third, to improve PCP’s ability to track referrals, subspecialty schedulers sent electronic notifications to PCPs regarding the final scheduling outcome: appointment scheduled, family not reached, or family declined. We anticipated that the most immediate measurable impact of these interventions would be an increased proportion of referrals resulting in a scheduled appointment. We recognized that referral scheduling would still be limited by inability to reach the family, lack of appointment availability, and families declining to schedule appointments. The downstream outcome we hoped to improve was visit attendance, although additional barriers external to the intervention could still prevent children from attending a scheduled appointment (ie, scheduling conflicts, financial/insurance barriers, transportation barriers, or resolution of symptoms). All CHP specialty services began using electronically transmitted referrals, including surgical subspecialties, medical subspecialties, and allied health professions (eg, audiology, physical therapy). Because this process was not designed to address urgent subspecialist care need, the process for obtaining urgent subspecialist advice or appointments still involved the PCP calling the subspecialist clinic or paging the subspecialist on call.

Electronically transmitted referrals were implemented at 15 PCP practices on September 9, 2015 (“pilot practices”). The remaining 24 affiliated PCP practices (“control practices”) continued the previous referral processes. Pilot practices were chosen among volunteers on the basis of practice size and engagement of physicians and clinic leadership; all 15 were within 25 miles of CHP. Control practices included practices that were and were not interested in participating in the pilot, and 12 of the 24 control practices were within 25 miles of CHP. After Plan–Do–Study–Act cycles within pilot practices, the intervention began at control practices from February 8, 2016, through May 3, 2016. Electronically transmitted referrals were live in half of control practices by April 1, 2016, and in all practices after May 3, 2016. Modifications over time included adding a field for preferred language and revising scheduler work processes.

The number of specialist physicians at CHP was relatively stable throughout the study period, increasing by 2.3% from April 2015 to April 2016, and by an additional 6.2% from April 2016 to September 2016.

Study of Intervention and Measures Chosen

The goal of this quality improvement evaluation was to assess processes and outcomes (Table 1) before and after electronically transmitted referrals. We linked administrative data from referring practices (ie, EMR referral orders and electronically transmitted referrals) to administrative data from subspecialty services to examine appointment scheduling and attendance. Our primary outcome was percentage of referrals followed by a subspecialty visit within 4 weeks of referral. This measure was chosen because it could be objectively measured, it was determined specifically for the patients at involved practices, and it was available in a timely manner for rapid-cycle improvement. Visits attended within 12 weeks of referral was a secondary outcome, in part because some referrals may not require a visit within 4 weeks. We aimed for a modest 5% improvement in the percentage of visits attended within 4 and 12 weeks of referral. We chose this goal in recognition of system complexity, barriers that our intervention would not affect (ie, limited appointment availability, transportation difficulties), and potentially valid reasons for referral noncompletion (ie, resolution of symptoms, seen by outside provider).

Table 1.

Process Measures, Outcome Measures, and Balancing Measures by Data Source

Data Source	Outcome Measure	Process Measure	Balancing Measure
Primary Measures
Administrative data (collected continuously)	Percentage of referrals with visits completed within 4 and 12 wk (goal: 5% increase)	Percentage of referrals with appointment scheduled within 4 and 12 wk (goal: 5% increase)	Percentage of referrals with scheduled within 4 wk with appointment nonattendance visit (goal: no change) Volume of referrals (goal: no change)
Complementary Measures
Survey data (collected before pilot and after full intervention)	PCP perceived ability to obtain timely routine subspecialty care (goal: 10% increase)	PCP satisfaction with scheduling process (goal: 10% increase) PCP ability to track referrals (goal: 10% increase)	PCP perceived ability to obtain timely urgent subspecialty care (goal: no change) PCP perception of quality of subspecialty care (goal: no change)

PCP indicates primary care physician.

Using administrative data, we also monitored process measures (percentage of referrals with an appointment scheduled within 4 and 12 weeks of referral) and balancing measures (percentage of scheduled referrals with nonattendance or no-shows; volume of referrals).

To complement this evaluation, we surveyed 190 PCPs at all 39 practices before the pilot and 6 months after full implementation using a 17-question Web-based survey distributed to an e-mail list. The survey assessed PCP perceptions of care processes (workflow, scheduling, referral tracking); outcomes (timeliness of routine subspecialty care); and balancing measures (quality of care; timeliness of urgent subspecialty care) via Likert-scale items and free-text comments. Approximately 50 schedulers were also surveyed 6 months after full implementation with a 9-question survey regarding workflow, information transmitted, and satisfaction with processes.

Analysis

To assess the impact at pilot practices, we used statistical process control methods to monitor rates of visits attended, appointments scheduled, and appointment nonattendance. Control charts were created by SPC for Excel (BPI Consulting, Cypress, Tex), using a shift of ≥7 points above centerline to identify special cause. To compare specific processes, outcomes, and balancing measures at pilot practices before and after pilot intervention, we also used chi-square tests. To further examine our primary outcome of visits attended within 4 weeks, we used interrupted time-series analysis,¹⁵ which fits separate linear regression models for each time period and tests for significant differences in intercepts and slopes using Wald t tests. To compare change in visits attended within 4 weeks at pilot versus control practices, we used difference-in-differences analysis with data from April 1, 2015, until February 7, 2016 (when control practices began receiving electronically transmitted referrals). For difference-in-difference analysis, we first confirmed that trends for pilot and control practices were not significantly different during the period before the pilot by estimating a linear regression model for visits attended within 4 weeks of referral using an interaction term between week and pilot practice status. We then estimated a linear difference-indifferences model for visits attended within 4 weeks of referral using a binary interaction term between study period (before the pilot vs pilot) and practice status (pilot vs control), with clustering by PCP practice. Finally, using chi-square tests, we performed post hoc analysis in which we limited our sample to practices with stable referral volume.

To measure the impact during subsequent implementation at control practices, we compared outcomes, processes, and balancing measures among control practices before and after full implementation using statistical process control methods and chi-square tests. For control practices, we defined the postimplementation period starting April 1, 2016, when over half of control practices had electronically transmitted referrals. We performed post hoc stratified analysis at control practices by distance between PCP practice and referral center (ie, <25 miles, >25 miles) using chi-square tests.

To assess change in PCP perceptions, we used chi-square tests to compare pre- and postsurvey results and also synthesized themes in the free-text comments.

Ethical Considerations

This quality improvement project aimed to improve patient care within our institution and was approved by the UPMC Quality Improvement Review Committee. Projects approved by this committee do not meet the federal definition of human subjects research, so formal approval by an institutional review board was not required.

Results

Across all practices, 33,485 unique referrals were entered during the study period (7770 before the pilot, 11,776 during the pilot, and 13,939 after full implementation; Table 2). Referrals per week increased over time, most notably for referrals during office visits and referrals to physician specialists. Visit volume across all 39 referring practices was relatively stable during this time; the total number of visits in April 2016 was 2.8% higher than April 2015, and number of visits in September 2016 was 4% higher than September 2015. The proportion of referrals for children of different ages was stable.

Table 2.

Characteristics of Referrals Orders During Each Project Phase

Characteristic	Phase
	Before Pilot	Pilot	Full Implementation
Start and end dates	April 1, 2015–September 8, 2015	September 9, 2015–March 31, 2016	April 1, 2016–September 30, 2016
Duration	23 wk	29.3 wk	26.1 wk
N	7770	11,776	13,939
N per week	338	402	533
Patient age
0–2 y	2320 (30%)	3701 (31%)	4203 (30%)
	101/wk	126/wk	161/wk
3–5 y	1623 (21%)	2337 (20%)	2560 (18%)
	71/wk	80/wk	98/wk
6–11 y	1982 (26%)	2883 (24%)	3586 (26%)
	86/wk	98/wk	137/wk
12–17 y	1635 (21%)	2585 (22%)	3149 (23%)
	71/wk	88/wk	120/wk
18+ y	210 (3%)	270 (2%)	441 (3%)
	9/wk	9/wk	17/wk
Practice
Pilot	3155 (41%)	5164 (44%)	5284 (38%)
	137/wk	176/wk	202/wk
Control	4615 (59%)	6612 (56%)	8655 (62%)
	201/wk	226/wk	331/wk
Referral placed during:
Office visit	4542 (58%)	7618 (65%)	9043 (65%)
	197/wk	260/wk	346/wk
Telephone encounter	2740 (35%)	3510 (30%)	3849 (28%)
	119/wk	120/wk	147/wk
Other	488 (6%)	648 (6%)	1047 (8%)
	21/wk	22/wk	40/wk
Requested specialty
Surgical	2081 (27%)	3701 (31%)	4601 (33%)
	90/wk	126/wk	176/wk
Medical	1598 (21%)	3016 (26%)	4051 (29%)
	69/wk	103/wk	155/wk
Allied health professionals*	4091 (53%)	5059 (43%)	5287 (38%)
	178/wk	173/wk	202/wk

Data are presented as n (%), number per week. During the prepilot phase, none of the practices had access to electronically transmitted referrals. During the pilot phase, only pilot practices had access to electronically transmitted referrals. During the full implementation phase, pilot and control practices had access to electronically transmitted referrals.

^*Allied health professionals included physical therapy, occupational therapy, audiology, and speech–language pathology.

Pilot Intervention

At pilot practices, the percentage of referrals with a visit attended within 4 weeks and 12 weeks demonstrated special cause with a shift of ≥7 points above the centerline after electronically transmitted referral implementation (Fig. 2A). Referrals with a visit attended within 4 weeks increased from 10.9% before the pilot to 20.0% (P < .001 in bivariate analysis), and within 12 weeks increased from 19.6% to 34.2% (P < .001). After implementation, 58% of electronically transmitted referrals specified intended completion at CHP. (Intended site of completion was not specified before the intervention.) For these referrals, postimplementation visit attendance was 29.5% within 4 weeks and 51.4% within 12 weeks.

Figure 2.

Percentage of visits attended within 4 weeks of referral at pilot and control practices. P charts are shown for proportion of referrals with appointment attended within 4 and 12 weeks of referral at (A) pilot practices and (B) control practices. Bold arrow indicates when electronically transmitted referrals were implemented at pilot practices. Gray ramp indicates time period over which electronically transmitted referrals were implemented at control practices. Red lines indicate upper and lower control limits. (For interpretation of the references to color in this figure legend, the reader is referred to the Web version of this article.)

Interrupted time-series analysis confirmed a statistically significant change in the percentage of visits attended within 4 weeks of referral (P < .001). There was no significant change in slope after implementation (P = .09; Fig. 3A), but the slope trended downward after implementation, implying reduced effect over time.

Figure 3.

Interrupted time-series analysis of visits attended within 4 weeks of referral. Interrupted time-series results for visits attended at (A) pilot and (B) control practices before and after electronically transmitted referral implementation at pilot practices (first dashed vertical line) and control practices (second dashed vertical line).

In difference-in-differences analysis, there was an 8.6% improvement in visits attended within 4 weeks of referral at pilot practices relative to control practices after pilot electronically transmitted referral implementation (95% confidence interval 3.9–13.3; P = .001).

Regarding processes of care, the percentage of referrals with an appointment scheduled within 4 and 12 weeks of referral demonstrated special cause after electronically transmitted referral implementation (data not shown). Percentage of referrals with appointment scheduled within 4 weeks of referral increased from 12.9% before the pilot to 24.1% (P < .001), and within 12 weeks from 23.5% to 42.4% (P < .001).

Among balancing measures, we observed no increase in percentage of scheduled referrals with visit nonattendance (9.3% vs 8.5% nonattendance among appointments scheduled within 4 weeks of referral; P = .73). Volume of referrals from pilot practices did increase (137 referral orders per week before the pilot to 176 referral orders per week during the pilot). Across practices, the median increase in referrals per week was 4 (interquartile range 1.5–8). To assess whether this increase in volume biased our results, we repeated analysis limited to pilot practices with stable referral volume (n = 6 practices; 71 referrals per week before the pilot and 74 referrals per week after the pilot), and observed similar improvement in visits attended within 4 weeks of referral, increasing from 11.8% before the pilot to 21.1% (P < .001).

Expansion of Intervention to Control Practices

On the basis of pilot results, electronically transmitted referrals were implemented at the remaining 24 practices from February 8, 2016, through May 3, 2016. The percentage of referrals from control practices with a visit attended within 4 weeks increased significantly but more modestly, from 11.8% to 14.7% (P < .001; Fig. 2B). Within 12 weeks, visits attended increased from 20.1% to 28.4% (P < .001). Among the 49% of electronically transmitted referrals specifying intended completion at CHP, visit attendance was 25.7% within 4 weeks and 50.8% within 12 weeks.

Interrupted time-series analysis demonstrated a significant change in intercept coinciding with implementation at control practices (P < .001; Fig. 3B), and no intercept change at the time of prior implementation at pilot practices (P = .74).

Similarly, percentage of referrals with a scheduled appointment within 4 weeks increased from 14.3% to 17.8% (P < .001). Percentage of scheduled referrals with visit non-attendance did not change (7.6% vs 6.5%; P = .25). Volume of referrals increased from 201 referrals per week before the pilot to 331 referrals per week after full implementation, with median increase across practices of 4 referrals per week (interquartile range 1–7).

To investigate this more modest effect at control practices, we conducted a stratified analysis by distance to academic children’s hospital. All pilot practices were located within 25 miles of the children’s hospital, but control practices included practices both within 25 miles and farther away. After stratifying by distance, control practices within 25 miles of the children’s hospital experienced a 5.8% increase in visits attended within 4 weeks, while control practices more than 25 miles away experienced a 0.1% decrease (Supplementary Table). Compared to control practices within 25 miles, control practices more than 25 miles away were less likely to refer specifically to CHP, and these more distant practices also had lower visit attendance rates even among referrals specifically to CHP (Supplementary Table).

PCP Surveys

Referring PCPs were surveyed before the pilot (n = 38/190) and 6 months after full implementation (n = 60/190). In open-ended responses, PCPs before the pilot discussed significant concerns about scheduling processes, appointment availability, and difficulties meeting urgent subspecialty care needs. After full implementation, open-ended responses praised the new referral process, voiced new concerns about referral tracking, and noted persistent issues with appointment availability and urgent subspecialty care. In quantitative results, the percentage who agreed or strongly agreed that they were satisfied with timeliness of subspecialty appointments increased modestly, from 3% to 17% (P = .002). The percentage who agreed or strongly agreed that referral processes were “good” increased from 18% to 62% (P < .001), with the greatest sources of dissatisfaction being appointment availability, referral tracking, and ability to access urgent subspecialty care.

Scheduler Surveys

In scheduler surveys 6 months after full implementation (n = 23/50), the percentage agreeing or strongly agreeing that the process was efficient and that the process was “good” were 35% and 30%, respectively. Many schedulers noted issues with the information transferred: only 17% agreed or strongly agreed contact information in electronically transmitted referrals is always correct, and only 13% agreed or strongly agreed reason for referral is always clear.

Discussion

The introduction of electronically transmitted referrals was associated with modest improvements in appointments scheduled and visits attended after referral, as well as increased PCP satisfaction with aspects of the referral process. Eliminating some of the scheduling barriers through electronically transmitted referrals succeeded in connecting more children to subspecialty care, and we continue to use the system on the basis of these findings. However, even with electronically transmitted referrals, only 1 in 5 referrals resulted in a visit within 4 weeks, and only 3 in 5 PCPs agreed that the referral process was “good.” Thus, while improvements were observed, the sizable percentage of children without completed referrals, the muted effect at control practices, and PCP surveys indicate that additional work is needed to address barriers to pediatric subspecialty care.

One opportunity is to improve the effectiveness of electronically transmitted referral system. First, scheduler surveys indicated difficulty contacting families, in part because of inaccurate patient contact information, which could be addressed by adding alternative telephone numbers or best times to call. Using other means of contacting families (ie, texting, patient portals) may also improve ability to contact families. Second, the current system does not capture the perceived urgency of referral, which could facilitate more responsive scheduling, although it still may not be able to accommodate urgent referrals, which remain a concern for many PCPs. Third, PCPs reported ongoing difficulty tracking referrals. While the electronically transmitted referral process originally included messages from schedulers to PCPs about the outcome of scheduling attempts, PCP survey responses indicated the need for better systems to track and manage referrals. Fourth, the improvements in timely appointments appeared to diminish over time at pilot practices, suggesting a need to strengthen maintenance of the intervention. On the basis of conversations with schedulers, we do not believe this was due to a loss of fidelity with the process. Another possible explanation for decreased effectiveness over time is that the cumulative success of electronically transmitted referrals decreased the availability of timely appointments in later months, especially after implementation at additional sites. Because appointments scheduled from the referring practices increased while the number of specialists increased only minimally, demand may have outpaced supply. Thus, addressing capacity constraints may be needed to support sustained effectiveness.

Impact could be further improved by increasing electronically transmitted referral use. We found that 7% of referrals that did not specify CHP still had a subspecialty visit at CHP, suggesting that PCPs may not be using the system for all relevant referrals. Increasing adoption of electronically transmitted referrals may require additional outreach to and feedback from PCPs. Second, the impact of electronically transmitted referrals could be enhanced by expanding its reach to additional PCPs and non-CHP subspecialists. PCPs >25 miles more frequently referred to non-CHP subspecialists, suggesting they might particularly benefit from a more comprehensive network of electronically transmitted referrals. Incorporating additional practices and subspecialists would require alternative electronic platforms, but it has the potential to more widely improve access to timely subspecialty care.

In addition to building on this initial improvement by optimizing effectiveness, adoption, and reach, we must also address barriers not targeted by electronically transmitted referrals, such as appointment availability and transportation barriers. While electronically transmitted referrals do not address these barriers, electronically transmitted referrals do allow for improved assessment of barriers and unmet demand. This in turn may help guide additional interventions, potentially including use of telemedicine to address geographic barriers,¹⁶ use of e-consultations to triage referrals,¹⁴ or incorporation of advanced practice providers¹⁷ or general pediatricians¹⁸ into subspecialty practices to expand capacity.

Our analysis is limited by available data. First, electronically transmitted referrals did not capture the urgency or necessity of referrals, and our data did not include patient insurance status, race/ethnicity, or other socioeconomic variables, limiting further subgroup analysis. Second, as noted above, the field specifying whether a referral should be completed at CHP did not exist in the preintervention order, necessitating that our analysis include all referrals (rather than only CHP-specific referrals). We also did not have data on utilization outside of our health care system, such that some referrals without visits may represent needs met elsewhere rather than unmet need. These limitations potentially increased the denominator and reduced the numerator for visit attendance, respectively, decreasing the absolute rates of visits attended in our primary measures. However, we do note that among postimplementation electronically transmitted referrals specifically to CHP, rates of visits attended within 12 weeks were within the range of referral completion across specialties (27–90%) in the prior work.¹⁹ Third, we noted increased referral volume after implementation. This increase was beyond what would be expected from a minimal increase in volume at referring practices. One possible explanation is that not all referrals were documented via EMR orders before implementation. To examine whether increased referral volume biased our results, we examined the impact among pilot practices with stable referral volumes and found results similar to our main results. Finally, we did not directly survey families about their perspectives during this evaluation. However, there was an improvement among CHP patients on the CAHPS Clinician and Group survey regarding subspecialist access (increasing from 52.3% to 76.2% of patients reporting maximum satisfaction with the access domain), but these data capture the experience of patients who actually attended a visit with CHP subspecialists, as opposed to referred patients, and so were not the focus of our analysis. Additionally, PCP survey response rates were low, but we included the PCP and scheduler survey results to provide additional perspectives to complement our more robust primary evaluation.

Despite these limitations, our analysis demonstrated incremental improvement in subspecialty visits attended after referral with implementation of electronically transmitted referrals between an academic children’s hospital and affiliated PCP practices. While our evaluation demonstrated improvements, our evaluation also indicates ongoing barriers to pediatric subspecialty care. Our results also suggest potential next steps, including increasing electronically transmitted referral effectiveness, adoption, and reach while also using electronically transmitted referral data to better understand and target additional barriers.

What’s New.

Implementation of electronically transmitted referrals resulted in incremental improvement in subspecialist visit attendance after referral, increased appointment scheduling after referral, and no increase in no-show rates.