Electronic consultations (E-consults) and their outcomes: a systematic review

Abstract

Objective

Electronic consultations (e-consults) are clinician-to-clinician communications that may obviate face-to-face specialist visits. E-consult programs have spread within the US and internationally despite limited data on outcomes. We conducted a systematic review of the recent peer-reviewed literature on the effect of e-consults on access, cost, quality, and patient and clinician experience and identified the gaps in existing research on these outcomes.

Materials and Methods

We searched 4 databases for empirical studies published between 1/1/2015 and 2/28/2019 that reported on one or more outcomes of interest. Two investigators reviewed titles and abstracts. One investigator abstracted information from each relevant article, and another confirmed the abstraction. We applied the GRADE criteria for the strength of evidence for each outcome.

Results

We found only modest empirical evidence for effectiveness of e-consults on important outcomes. Most studies are observational and within a single health care system, and comprehensive assessments are lacking. For those outcomes that have been reported, findings are generally positive, with mixed results for clinician experience. These findings reassure but also raise concern for publication bias.

Conclusion

Despite stakeholder enthusiasm and encouraging results in the literature to date, more rigorous study designs applied across all outcomes are needed. Policy makers need to know what benefits may be expected in what contexts, so they can define appropriate measures of success and determine how to achieve them.

INTRODUCTION

Health care systems face increasing pressure to manage costs while assuring access to care. Electronic consultations (e-consults) are increasingly used as one way to approach this challenge. An e-consult is an asynchronous consultative communication between clinicians occurring within a shared electronic health record (EHR) or secure Web-based platform. E-consults offer a way for referring clinicians (usually primary care providers [PCPs]) to obtain rapid input from specialists and often obviate the need for a face-to-face patient visit. Benefits include low direct costs to the health care system, reduced care fragmentation, avoided travel for patients, and a platform for documented clinical communication between clinicians. E-consults are postulated to improve access to care by creating availability of specialty care services where once they were limited or absent; facilitating timeliness of specialist input for patients served by e-consultation; and reducing wait times for face-to-face visits by reducing the pool of patients who require them.

In a 2015 systematic review,¹ we reported that most of the peer-reviewed research on e-consults originated from within 3 major integrated North American health care systems, each with different organizational and financial structures. We found that e-consults are flexible in their application, highly useful to referring clinicians, and improve timeliness of specialty advice. We also identified several gaps in the early literature, most notably a paucity of objectively measured outcomes, which are essential to policy-makers assessing the potential adoption of e-consult programs. Despite this limited evidence base for outcomes, e-consults have continued to spread more broadly within the US and internationally across a range of single-payer systems, accountable care organizations, and fee-for-service models.

The focus of this systematic review is to summarize the peer-reviewed literature on outcomes of e-consult programs since our 2015 review. Our focus is on the outcomes that are central to the provision of high-quality care. As described by the Quadruple Aim framework, these outcomes are population health, reducing per capita health care cost, and enhancing both the patient and clinician experience of care. Access to health care is a prerequisite for the 3 patient-focused outcomes of the Quadruple Aim, and a primary goal of e-consultation, so we have addressed this outcome as well. We aim to answer 3 questions: 1) What is known about the relationship between e-consults and access to care? 2) What is known about the relationship between e-consults and other important outcomes: cost, quality of clinical care, and patient and clinician experience? 3) What are the critical gaps in existing research on e-consult outcomes?

METHODS

We conducted a systematic review of this topic, including use of a formal search strategy, an appraisal of study quality, and a narrative synthesis of findings.² We completed a narrative rather than a quantitative synthesis because our questions of interest are broad and the relevant studies are heterogeneous.³ To inform our review, we performed a systematic search of 4 databases (PubMed, CINAHL, Embase,⁴ and Cochrane Library) for articles about e-consults published in English between January 1 2015 and February 28 2019, using keywords including e-consult, econsult, electronic consult, and eReferral. We then searched reference lists of included articles for relevant studies. (See Supplementary Appendix Exhibit 1 for search terms and strategy.)

As per our 2015 review, we defined an e-consult as “an asynchronous consultative communication between clinicians occurring within a shared EHR or secure Web-based platform.” E-consults can serve as the entry point to all referral encounters (eg, the Champlain BASE program [Ontario, Canada]⁵ or the San Francisco [California] Health Network⁶) or as one type of referral option that referring providers can choose at their discretion (eg, the US Veterans Health Administration [VHA]⁷). E-consult platforms may be in the form of a web portal (eg, Champlain BASE) or a shared medical record (eg, VHA). We included empirical studies that reported on outcomes related to access, cost, care, or patient or clinician experience. We excluded articles about electronic referral only; patient-provider modalities; other forms of consultation such as video, telephone, or email; and interventions that bundled e-consults into multicomponent initiatives or programs and did not measure the effect of e-consults separately. Working in pairs, the investigators reviewed the title and abstract of each potentially relevant article. We used group discussion to reach consensus when there was disagreement about relevance. For each relevant article, one investigator abstracted information on its design and findings, and the work was confirmed by a second investigator. We summarized the results of relevant studies, organized by our 3 research questions for this review.

To assess the strength of evidence for each outcome, our group applied the Cochrane GRADE (Grading of Recommendations Assessment, Development, and Evaluation) rubric.^8,9 GRADE is a systematic approach to rating the evidence in systematic reviews and is particularly useful for rating the quality of evidence across multiple outcomes. GRADE is subjective but uses transparent criteria. Using GRADE, reviewers evaluate the quality of evidence for outcomes reported in systematic reviews on a 4-point scale from very low (+) to high (++++).¹⁰ A very low score means that the true effect is probably markedly different from the estimated effect; a low score means that the true effect might be markedly different from the estimated effect; a moderate score means the true effect is probably close to the estimated effect; and a high score means that the reviewers have confidence that the true effect is similar to the estimated effect. Using the GRADE approach, observational studies without special strengths or important limitations provide low-quality evidence and randomized trials with few limitations provide high-quality evidence. Ratings can be modified down based on the risk of within-study bias, inconsistency, indirectness of evidence, imprecision, and publication bias. Infrequently, evidence from very strong observational studies may be upgraded in the face of large effects, dose–response gradients, and a lack of obvious bias. For each outcome, the weight of the evidence across relevant studies provides support for a final score for the outcome. For this review, one investigator reviewed the evidence to assign GRADE scores, and the research team assessed the evidence for each outcome to arrive at a final score for each.

RESULTS

Overview

The database search identified 1544 total publications. Details of study selection are presented in the PRISMA diagram online (Supplementary Appendix Exhibit 2ⁱ). Characteristics of the 63 selected articles are provided in the Supplementary Appendix table online (Table 1) (Appendix Exhibit 3ⁱ). Studies are listed by outcome in the Supplementary Appendix online (Table 2) (Appendix Exhibit 4ⁱ).

More than half of the included studies were from 1 of 2 settings: 26 from the Champlain BASE program^11–36 and 13 from VA settings.^7,37–48 Of the remaining 24 studies, there were 12 from community health centers or large public health care systems,^43,49–59 8 from academic medical centers,^60–67 2 from a nongovernmental humanitarian organization,^68,69 1 from a not-for-profit integrated health care delivery system,⁷⁰ and 1 from a private company providing e-consult services.⁷¹

Most studies were observational in design and lacked a control group. One study reported data from a pilot randomized controlled trial;⁵³ 3 studies reported results from cluster randomized controlled trials;^49,56,70 3 studies were pre-post evaluations with a comparison group;^62,66,72 and 1 study employed a retrospective cohort design with a comparison group.⁴⁸ Most results were positive. Below we present an overview of the GRADE scoring assessment, summaries of the evidence for each outcome, and the rationale for the GRADE scores assigned to the quality of evidence for each of those outcomes.

GRADE scores

At baseline, GRADE scores were low for all outcomes, because all but 4 of the 63 studies were observational in design and most of those lacked a control group. There was no outcome for which the weight of the evidence demonstrated the large effects, dose-response relationship, and lack of obvious bias that could lead to an increase in the baseline GRADE score for that outcome. The GRADE score for quality of clinical care and patient experience was downgraded to very low after assessment of criteria for score adjustment. The preponderance of positive studies across all outcomes raised concern for publication bias.

Access to care

New access to specialist input

Since 2015, articles have been published that extend e-consult to patients that might not have had access to a specialist face-to-face visit in the first place, such as patients with rare conditions⁴⁶ and patients in rural,⁴² extremely remote,²⁷ or otherwise low-resource regions.⁶⁸

Faster access to specialist input compared to face-to-face visits

Sleep medicine e-consultation followed by the specialists’ order of a sleep study and positive airway pressure therapy, if indicated, resulted in a decrease over 4 years in the interval between sleep consult and positive airway pressure prescription from ≥ 60 days to ≤ 7 days.³⁸ More than a dozen studies report a response time (i.e., the time between when the e-consult is requested and when the responding specialist provides the response) of a few days or less,^{11,13,14,16,20,21,28,32,34,35,40,46,47,51,56,61,62,67,69} whereas face-to-face visits generally require days to weeks to be completed.

Reduced wait times for face-to-face clinic visits

We identified 2 published studies demonstrating reduced wait times for face-to-face clinic visits,^51,62 and 2 demonstrating no change.^38,58 The studies that reported improved wait times were observational studies in large e-consult programs. In the first study, using a pre-post design, researchers observed a decrease in wait times soon after e-consultation implementation at a multi-site academic medical center in a Medicaid-funded fee-for-service model. The number of referrals for face-to-face visits decreased by 19% and the proportion of face-to-face referrals seen within 14 days increased from 29% to 35% from the baseline year to the intervention year.⁶² In the second study, which also employed a pre-post design, wait times for face-to-face visits across specialties in the Los Angeles Department of Public Health (LADPH) system decreased by 17% 3 to 4 years after e-consult implementation.⁵¹

Specialist input on questions that might not otherwise have warranted a face-to-face referral

The relationship of e-consults to avoided face-to-face referrals is not one-to-one. In a single-center VA study of e-consultation to an infectious disease service, the total number of face-to-face consults remained steady after implementation of the e-consult program, even though e-consult volume far exceeded that of face-to-face consults.⁴⁷ The authors observed that e-consults were being used to ask about topics such as positive urine cultures and travel immunizations that were not previously referred for face-to-face visits. In a study of nephrology e-consults at an academic tertiary care center, PCPs responded to a brief survey after each e-consult.⁶³ If e-consults did not exist, in 69% of cases the PCP would not have sought specialty consultation via email or telephone. A study in a VA hematology unit found that the annual number of face-to-face visits decreased 18% within 2 years after e-consult implementation, but the total number of consults (e-consult plus face-to-face) increased by 80%.⁴¹

Though many in number and consistent in suggesting a benefit, studies informing the access outcome are all observational. Among the studies of wait times that used a pre-post design, none included a control group. This limits the ability to attribute improvements in wait times to e-consult implementation rather than to other organizational changes or secular trends. The GRADE score for the outcome of access therefore was not changed from the baseline score of low.

Costs

Costs to payers

Payer costs have been explored in studies using administrative data. In a pre-post study in an academic fee-for-service setting, Gleason et al reported that introduction of an “e-consult first” program was associated with a 7.25% decrease in total professional fees (ie, charges for direct care by the provider) in the 120-day period following the specialty contact.⁶² A limitation of this study is the inability to attribute the lower costs to decreased utilization by patients who had an e-consult, since all patients received an e-consult first, but some were recommended for a face-to-face visit after specialist review.

Anderson et al conducted a cost-effectiveness analysis for Medicaid-insured patients in a cluster-randomized trial of cardiology e-consults for all nonurgent referrals versus the traditional face-to-face consultation process in a statewide federally qualified health center.⁴⁹ Using claims data, the authors calculated that total per-patient costs to the payer and costs of cardiac tests and procedures were lower among patients receiving care in the e-consults arm by $466 USD and $81 USD, respectively. In in the same health care system, a separate observational study reported average costs per month that were $82 lower per patient across 4 medical and surgical specialties after the introduction of e-consults.⁵⁰ While specific dollar figures are difficult to interpret and generalize to other health care systems, these 3 studies are at least consistent with the hypothesis that e-consult programs are generally cost-saving to payers.

Costs to patients

Costs to patients from travel were addressed in 3 studies. Kirsh et al calculated that VA patients served by e-consults in 2011–2013 would have had to travel an average of 72 miles to see a specialist face-to-face. The authors estimated $2.8 million USD savings attributable to avoided travel for the entire nationwide VA system.⁴² Lee et al⁷² calculated that a proactive fracture liaison service led to 69 miles of travel avoided per patient. A cost evaluation of the Champlain BASE Program in Canada demonstrated a net savings to society (from avoided referrals and patient travel/work absence) of $38 729 over 12 months, or $11 (CAD) per e-consult.²⁵

Cost savings by avoidance of face-to-face referrals

Several studies have taken indirect approaches to examining cost savings by examining the association of e-consults with face-to-face referrals. Two studies compared the change in face-to-face referral rates pre- and post-implementation between specialties with and without e-consultation.^62,66 One year after implementation of a cardiology e-consult program in an academic medical center, the increase in face-to-face cardiology referrals was less than that for specialties without e-consultation (4.5% vs 10.1%, P < .001).⁶⁶ E-consults in an academic medical center were associated with a decrease of 19% in face-to-face referrals, compared to an increase of 3.6% among nonparticipating specialties.⁶²

Several publications from the Champlain BASE program report on a PCP survey question at the end of each e-consult request about whether a face-to-face consult would have been sent in the absence of the e-consult option. Depending on specialty of the consultation being requested, 32%–80% of PCPs report that a face-to-face visit would have been required absent the e-consult,^{12–14,16,18–21,24,26,28–36,41,47,51,58,62,65} suggesting that up to as many as 4 out of 5 e-consults are perceived to obviate a face-to-face visit. Using these data on face-to-face referrals avoided, the Champlain BASE group calculated payer costs decreasing from $131.05 to $6.45 (CAD) per e-consult from year 1 to year 3 of the program.²³

Overall, a robust number of studies addressed costs, including 1 cluster-randomized controlled trial and 2 pre-post observational studies. These were consistent in their evidence for a positive impact of e-consults on costs. However, many studies of cost used an indirect approach to address cost savings by comparing volume of e-consults to face-to-face visits, or by asking the referring provider whether a face-to-face referral had been avoided. Therefore, the GRADE score for the outcome of cost remained low after assessment of the strengths and weakness of the body of evidence.

Quality and safety outcomes

Quality of clinical care

We identified 2 studies comparing clinical outcomes among patients managed by e-consult versus face-to-face referral. In a small pilot study, high-risk patients with chronic kidney disease (N = 23) were randomized to either e-consult or face-to-face referral; renal function was similar in both groups when assessed approximately 12 months after consultation.⁵³ A nonrandomized study of e-consults versus face-to-face referral for diabetes management at a single center found similar reductions in A1C at 6 months in unadjusted analyses.⁴⁸ In both of these studies, patient characteristics were similar between the e-consult and the control groups.

Four studies reported what may be called “intermediate” clinical outcomes. By “intermediate” we mean outcomes that represent clinical actions or process measures that are on the causal pathway to hard clinical endpoints of morbidity and mortality, but that may occur in a shorter timeframe and are often easier and less expensive to measure than those endpoints. In the first study, a proactive e-consult program for patients with recent fracture resulted in improved rates of osteoporosis medication provision and bone density testing. This program relied on a nurse liaison to identify fracture patients for e-consultation and then coordinate with PCPs to complete the recommended workup and support medication provision.⁷² In the second study, a clinician’s order of intravenous vancomycin triggered an e-consult to the infectious disease service, which then provided standardized prescribing guidance; this pre-post study reported an increase in rates of appropriate antibiotic prescribing, from 45% (71/146 patients) to 55% (51/98).⁴⁴ In the third study, an examination of e-consults for anesthesiology preoperative evaluation in one VA regional network, there was no increase in preventable operating room cancellations attributable to the use of e-consults.³⁷ A fourth study of e-consults for sleep medicine over 5 years reported a decrease in the interval between sleep consult and positive airway pressure (PAP) therapy prescription from ≥ 60 days to ≤ 7 days.³⁸

In sum, intermediate outcomes may improve when the responsibility for requesting or implementing recommendations from e-consults is automated or shared by the specialist service. Whether ultimate clinical outcomes are worse, better, or as good with e-consults remains unclear.

In assessing whether the GRADE score for clinical quality should be changed from the baseline score of low, we acknowledged that the literature on quality of clinical care was represented by just a handful of studies (N = 6) that provided only indirect evidence through assessment of intermediate clinical outcomes or process measures. Intermediate outcomes are often faster and easier to capture than hard endpoints, but the correlations between these intermediate measures and hard endpoints can be diluted by other factors. These limitations led to downgrading the GRADE score for clinical quality evidence to very low.

Safety outcomes

Several studies reported on adverse events or emergency department utilization following e-consultation. One cluster-randomized trial found no statistically significant difference in a secondary outcome of emergency department utilization between cardiology patients in the traditional consultation pathway and those in the intervention pathway, whose consults were triaged via e-consult to either completion via e-consult or face-to-face referral.⁵⁶ In a pre-post study, the proportion of patients with a specialty encounter (referral or e-consult) who had an emergency department (ED) visit within 120 days post-encounter decreased from 9.8% prior to e-consult implementation to 8.6% after e-consults became an option for PCPs.⁶² In a pre-post study of endocrinology e-consults, there was no change in the percent of consultations (face-to-face or e-consult) that were followed by an emergency department (ED) visit or hospitalization within 6 months.⁶¹ Several studies employing manual review of the medical record found no increase in adverse events (eg, ED visits, hospitalizations, and/or death within 1–6 months) following e-consult.^{40,55,60,64–66} Among 500 patients referred for cardiology e-consults, 2.2% required hospitalization related to the problem addressed in the e-consult within 6 months; no comparable rate of hospitalization after face-to-face consults was provided.⁴⁰ A study of surgical e-consults reported that 4 of 36 patients who did not have clinic visits following e-consultation had ED visits for problems related to the initial reason for referral.⁵⁹ In comparison, among 114 patients scheduled for face-to-face surgical consultation, 4 had ED visits before their surgery clinic appointment.

Taken together, these studies suggest that e-consults do not appear to greatly increase the risk of sentinel events, perhaps because sicker patients are more likely to receive face-to-face referrals instead. With 1 exception,⁶² the observational studies we examined that reported on safety outcomes did not include data on the characteristics of patients who ultimately received e-consults vs face-to-face referrals, limiting the ability to judge the extent of this potential bias. Neither the cluster-randomized trials nor the pre-post studies defined adverse events as a primary outcome and were therefore most likely underpowered to draw definitive conclusions about risk.

We found no studies of less severe but nonetheless important safety outcomes, such as specialist or PCP errors in medication recommendations or patient errors in medication-taking. Such mistakes may be more likely given that patients in an e-consult situation are not directly involved in the clinical conversation. Without having the patient in the room, specialists may not be able to ensure medication reconciliation or patient understanding of instructions; medication errors or inadequate communication could cause patient harm without resulting in ED visit or hospitalization.

While studies of safety were consistent in finding no increased risk with e-consults, underpowered analyses and risk of bias limited the certainty of evidence from those studies, and so the GRADE score for the overall quality of evidence on safety remained low.

Patient and clinician experience

Patient experience

As with our previous literature search, we found studies that report high levels of patient satisfaction with e-consults as measured by surveys.^41,60 Benefits elicited in qualitative interview studies with patients include faster access to care, avoidance of travel for face-to-face referrals, and strengthening of the role of primary care providers.^15,45,57 Studies reveal a trade-off in patient experience: though patients may agree with the PCP’s suggestion to send an e-consult and reap benefits of faster access, they also lose the opportunity to ask the specialist questions that would have arisen during the course of a face-to-face visit. Moreover, e-consults do not allow patients to participate in a conversation with the specialist about treatment that accounts for patients’ preferences and values. One study specifically examined this issue, finding that the lost opportunity is a drawback that patients may accept in return for the benefits of e-consults, but only in certain clinical situations or when the wait for a face-to-face appointment is very long.¹⁵

The implications of the patient’s exclusion from the clinical conversation have not been extensively explored. One qualitative study found that, in general, patients were comfortable with a minimal level of direct engagement with e-consults, but wanted to be informed about the process and wanted assurance that their history and concerns would be comprehensively communicated between the PCP and specialist.⁵⁷ Patients reported that a strong relationship with their PCP helped them feel more comfortable with receipt of virtual specialty care.

Overall, data on the patient experience indicate that patients are willing to accept the drawbacks of e-consults when the risks of not participating in conversation with the specialist are low and trust in their PCP is high, and when wait times are excessive. Aspects of experience outside of acceptability and satisfaction with avoided referrals have not been studied.

Overall, there were a very small number of studies on the patient experience—2 survey studies and 3 qualitative studies. The GRADE score is applicable to the quantitative survey studies. Because there were only 2 of these, thereby increasing concern for imprecision, the GRADE score for the quality of evidence of patient experience was downgraded to very low.

Primary care provider experience

At the time of our 2015 review, most studies reported high PCP satisfaction with e-consults, including perceived value, improved timeliness of specialty input, and enhanced communication with specialist. Newer studies using PCP surveys support these findings,^{11,13,14,18–22,24,26,29–31,33–36,41,54,57,60,62–65,67–71} and 2 qualitative studies also report generally high PCP satisfaction.^7,39 More recent studies have also described a strong educational benefit to less-experienced PCPs. In a survey of UCSF PCPs, for example, nurse practitioners, physician assistants, trainee physicians, and physicians with ≤ 10 years of experience were more likely to report improved clinical management skills than physicians with longer experience.⁵⁴ Another survey of PCPs in the Champlain BASE program found that nurse practitioners were more likely than family physicians to report that an e-consult modified the original planned course of action.²⁴ A cluster-randomized trial surveyed PCPs and found that those using psychiatry e-consults reported improved perceived support in management of mental illness, though not increased self-efficacy related to depression management.⁷⁰

Among more mature e-consult programs, studies leveraging qualitative methods have highlighted shortcomings for PCPs. In most health care systems described in the literature, PCPs are not provided additional time or compensation to manage patients served by e-consult. In a qualitative interview study, VA PCPs reported that e-consults increased workload for the PCP, who must decide whether and how to follow through on specialists’ recommendations, manage the condition, and follow up with the patient.⁴⁵ A qualitative interview study among Los Angeles Department of Health Services PCPs revealed mixed opinions about whether the burden incurred by managing patients via e-consult is worth the tradeoff in timeliness and ability to manage conditions.⁴³ Findings from a safety-net program in Denver, Colorado were similar.⁵²

Specialist experience

Specialists report ambivalence towards e-consult systems, a finding consistent with our earlier review. Surveys of specialists in 2 different US academic medical centers found that most report “optimal complexity”⁶² and that e-consults are efficient and help avoid referrals.⁶³ Concerns about e-consults relate to both organizational and clinical concerns. In 3 qualitative interview studies conducted within the VA (1 in anesthesiology, 2 across multiple medical specialties), specialists perceived value to patients, PCPs, and their own clinical practice.^7,39,45 However, dissatisfaction resulted from lack of or inadequate compensation or protected time, concerns about liability, and frustration with repeatedly answering the same questions. Conflicts with inpatient schedules and inadequate compensation were also raised on a survey of VA cardiologists.⁴⁰ Specialists in the Champlain BASE system rated e-consults high on a survey assessing feasibility of e-consults to improve access to care, impact on PCP-specialist communication, educational value for PCPs, and whether the program should be expanded; there were mixed opinions about appropriateness of the current compensation model in that¹⁷ as well as in another survey study among Ontario providers of e-consults from 2 different systems.¹⁹ Concerns about workload and insufficient reimbursement were echoed in survey data from a study among gastroenterologists in a US academic medical center.⁶⁵ In e-consult programs for inpatient care and for anesthesiology pre-operative assessment, and in the Denver safety-net multispecialty e-consult program, some specialists raised concerns about providing care without seeing the patient.^37,52,64 Specialists voiced a need for more consistent communication about the e-consult process.⁵⁷

Taken together, studies of referring and consulting clinicians suggest that the greatest barrier to clinician satisfaction is related to how e-consults are implemented and supported within the organization.

Survey results from PCPs and specialists were consistent and suggested strong benefits to the clinician experience, but inherent limitations of the study designs prevented the GRADE score for this outcome from being upgraded from low.

DISCUSSION

Our 2015 literature review found that few studies addressed the outcomes of e-consult programs. We conducted the current review to determine what progress has occurred since that time: a period that has witnessed considerable expansion of e-consult programs in health care systems across the US and internationally, the development e-consult business models, and a proliferation of literature.

We conclude that the empirical evidence for important outcomes of e-consultation remains modest. The overall quality of evidence across outcomes is generally low, since most studies are observational, resulting in persistent concern regarding the presence of bias and confounding. The rating of the quality of evidence for clinical care was downgraded to very low since all studies measured intermediate rather than hard clinical endpoints. The rating of the quality of evidence for patient experience was also downgraded on the basis of concern for imprecision. Because of heterogeneity among the main outcome measures used in these studies, it is difficult to compare the magnitude of effect across studies and challenging to assess the generalizability of findings. We note generally positive effects across studies for the subset of important outcomes that were measured, with mixed results for clinician experience. Though it is reassuring that the literature is generally positive, the preponderance of positive studies also raises concern about possible publication bias.

Stakeholder enthusiasm for e-consults is high, and the literature on outcomes to date is informative and encouraging, but many important questions remain for policy makers and health care system leadership. It is common for relatively low-quality evidence to support health system interventions, since randomized controlled trials are often cost-prohibitive, unethical, or not planned as part of implementation. However, some general principles can be applied to strengthen the evidence base for e-consults and support targeted improvement and cost-effective expansion. We recommend 3 major directions for future work: quasi-experimental designs to strengthen causal inference; application of comprehensive measurement frameworks that enable both overall assessment within and across e-consult programs as well as more precise assessments of specific outcomes; and longer follow-up periods to examine sustainability over time.

Quasi-experimental designs

E-consult outcome evaluation is well-suited to quasi-experimental designs, though few studies have taken this approach. In observational studies of e-consults, a major problem is that patients who specialists determine should receive e-consults are almost certainly unlike those who have either face-to-face consultations or no consultation at all. These differences, which epidemiologists would consider selection bias, can underlie whatever associations are found between e-consults and outcomes, such as cost and clinical outcomes of care. Any such associations may therefore reflect case-mix rather than a true causal relationship. Studies with comparison groups should report on demographic and clinical characteristics of the patients. Such data, as reported in some of the quasi-experimental studies we reviewed,^48,53,62 would reassure readers that the potential impact of selection bias is lessened. In systems where specialists make the final decision about whether patients are seen by e-consult or face-to-face visit, wide variation in individual specialists’ threshold to recommend referral^51,61 will also impact outcomes in the different groups. If quasi-experimental designs to estimate causal effects are not feasible, another possibility is a natural experiment design, in which costs and clinical outcomes among a cohort of patients with similar health profiles and a selected reason for referral is compared before and after implementation of e-consultation; such studies can capitalize on advanced statistical methods, such as propensity scores⁷³ and instrumental variables,⁷⁴ to strengthen the validity of observed results.

Comprehensive measurement

The overall value of e-consults to any health care system, as well as cross-system comparisons, will be difficult to assess by focusing on a single outcome measure. This is true for any complex intervention because gains in 1 outcome may be offset by losses in another. For example, e-consults increase access to specialist advice, but under what organizational and clinical circumstances does this lead to better quality of clinical care? Moreover, how does the increased role of the PCP in executing the specialists’ recommendations affect the patient-PCP relationship and the PCP’s work-life satisfaction and risk of burnout? Studies of clinical outcomes should aim to establish the causal pathway between e-consult and those outcomes; capturing clinical process measures in addition to outcomes can help establish the steps in the pathway. Examinations of safety should have control groups, account for clinical complexity, and measure less severe but still-concerning events. We also recommend cross-system evaluation of how clinician satisfaction, burnout, and quality of care are affected by different e-consult delivery models, processes, workflow accommodations, and compensation. There is a need for detailed and comprehensive economic evaluations that clearly describe program organization and account for both direct and indirect costs and savings. An e-consult that avoids a face-to-face specialty visit may lower per patient expenditure, but what happens to system-wide costs if 20%, 30%, or 40% of e-consult requests are for evaluations that are not actually needed?

Comprehensive assessment of e-consult benefit will require a broad set of measures that address the interests of key stakeholders that can be applied and compared across health care systems and that enable tracking of access, economic impact from multiple perspectives, clinical care, and patient and clinician experience over time. For example, 1 group proposes the RE-AIM framework for cross-system studies of e-consult programs.⁷⁵ Such evaluations would be enhanced by application of qualitative methods and implementation science frameworks that help illuminate how, why, and under what conditions e-consult programs are successful in achieving important outcomes.

Longer follow-up

Most pre-post or randomized studies that we identified had relatively short follow-up periods. Other studies with longer follow-up periods generally did not present trends over time. This shortcoming limits the opportunity to examine sustainability of e-consult outcomes. For example, as the backlog of pending consults decreases through resolution via e-consult, improvements in clinic wait times may plateau. Only 1 study reported on sustained improvements in clinic wait times years after e-consult implementation,⁵¹ but such impacts are likely to vary by system. Studies of access and wait times over longer follow-up periods will need to account for secular events such as staffing changes, organizational initiatives, and the increasing availability of multiple modalities of specialty care delivery, including telemedicine/video visits and secure messaging between clinicians and patients.

In summary, we found only modest evidence for e-consults’ impact on outcomes related to access, cost, quality of clinical care, and patient and clinician experience. Positive findings are generally reported among the diverse questions that have been addressed, but most study designs have been observational; cross-system empiric studies are lacking; and there is limited depth of understanding for many outcomes. E-consult spread has been rapid, with programs offered by an increasing number of nonprofit health care systems,⁷⁶ academic medical centers,⁷⁷ and private companies.^78,79 The Centers for Medicare & Medicaid Services began reimbursements for e-consults in 2019.⁷⁶ Yet research still lags behind enthusiasm of policy makers, health care systems leaders, and others involved in the implementation of new programs.

Many health care systems have turned to e-consult programs to improve specialty access without increasing costs. Existing data are promising and explain the continued growth of e-consult programs, but rigorous studies are needed to understand exactly what benefits can be anticipated in different organizational, clinician, and patient contexts. Policy makers, health systems leaders, and funders need this information to define reasonable measures of success for e-consultation and to plan appropriate improvements to achieve such measures.

FUNDING

This work was supported by a Department of Veterans Affairs Health Services Research & Development Career Development Award (grant number IK2HX002137). Disclaimer: The contents do not represent the views of the US Department of Veterans Affairs or the United States government.

AUTHOR CONTRIBUTIONS

All authors contributed to the analyses and interpretation of data, revised the work critically for important intellectual content, approved the final version to be published, and are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.