Abstract
Background
There have been no large-scale studies to date of patients’ experiences with electronic consultation (eConsult) between primary and specialty care.
Objective
Compare experiences with eConsult and referral for in-person specialist consultation.
Design
Online survey 2–6 weeks following eConsult or referral at 9 US academic medical centers.
Participants
Adult patients with no more than one eConsult or referral order from a primary care provider (PCP) in the prior month. Over 9 months, 29,291 email invitations were sent (88% referral; 12% eConsult).
Main Measures
Trust in and satisfaction with PCP; consult type awareness; agreement with decision to seek specialist input; timeliness of care; mode of PCP-patient eConsult communication; satisfaction with specialist’s recommendations; future preference for eConsult or referral.
Key Results
A 27.6% response rate yielded 8087 respondents (88.4% referral; 11.6% eConsult). Many did not know that their PCP had placed a referral (32.8% unaware) or eConsult (52.9%), and eConsult awareness was significantly higher among patients reporting better health (OR 1.62, 95% CI 1.18–2.23). Most (81.4% eConsult; 82.0% referral) were satisfied with the specialist’s recommendations. Those who had a good primary care experience were more likely to be satisfied (eConsult: OR 10.63, 95% CI 2.95–38.32; referral: OR 2.87, 95% CI 1.86–4.44). For a similar problem in the future, 78% of eConsult and 32% percent of referral patients preferred eConsult.
Conclusions
This multisite study demonstrates that many patients find virtual consultation to be an acceptable strategy for the management of their medical condition and that trust and confidence in one’s PCP are crucial ingredients for a satisfying eConsult experience. The lack of awareness of eConsult among many patients who were beneficiaries of the service warrants an increased effort to include patients in eConsult decision-making and communication. Further research is needed to assess eConsult acceptability and satisfaction in more diverse patient populations.
Electronic supplementary material
The online version of this article (10.1007/s11606-020-05703-7) contains supplementary material, which is available to authorized users.
KEY WORDS: eConsult, telemedicine, primary care, specialty care, access to care, patient preferences, patient satisfaction
INTRODUCTION
Electronic consultation (eConsult) services are emerging as a promising means of leveraging electronic health records (EHR) to improve inter-clinician communication and increase the timeliness and efficiency of medical care. eConsult enables primary care providers (PCPs) to request a virtual, asynchronous consultation with a specialist about a specific patient via a shared EHR. eConsult programs have been successfully implemented at a range of health care institutions in the USA, Canada, and Europe and are associated with reductions in traditional referral rates, shorter wait times for specialty care, and improved communication between clinicians.1–7
A growing body of literature documents PCP and specialist experiences with various eConsult models,8–12 but less attention has been paid to patients’ awareness of and experiences with eConsult. Studies to date have either drawn on providers’ impressions of patients’ satisfaction with eConsult or have a small sample size.13, 14 In conjunction with the national dissemination of a multi-specialty eConsult program originally developed at the University of California, San Francisco (UCSF) and supported by a Center for Medicare & Medicaid Health Care Innovation Award, this study aimed to (a) assess patients’ awareness of and satisfaction with eConsult and (b) compare the experiences of patients whose PCP used eConsult with those of patients who were referred for an in-person visit with a specialist.
METHODS
Study Setting
Participating health systems included nine academic medical centers that provide primary care and specialty outpatient medical services: University of California health system campuses at San Francisco, San Diego, Irvine, Davis, and Los Angeles; Dartmouth-Hitchcock, Lebanon, NH; University of Virginia, Charlottesville; University of Iowa, Iowa City; and University of Wisconsin, Madison. The eConsult model was launched at UCSF in 2012 and at the remaining institutions between 2014 and 2015. Patients receiving an eConsult at these participating institutions were not responsible for any costs for the service.
Instrument Development
To better understand and compare patients’ experiences with eConsult and traditional referrals, we developed a survey informed by 15 semi-structured interviews with primary care patients whose PCP had used the eConsult service at UCSF. Through these interviews, we found that many patients were not aware that an eConsult had taken place on their behalf. These findings prompted us to develop survey questions that did not depend on patients’ prior awareness of eConsult. The survey included branching logic to direct respondents depending on recognition of the PCP’s intention to place an eConsult or traditional referral order (see Fig. 1 for survey flow and online Appendix A for survey questions). Survey design was informed by a Kaiser Permanente study comparing patients’ satisfaction with virtual and traditional consultation14 and the Consumer Assessment of Health Plans Survey (CG-CAHPS).15 Question wording and order were adjusted based on pilot testing and cognitive interviews with 18 primary care patients at UCSF. The interviews assessed participants’ thought processes as they responded to survey questions, including how questions and key concepts were interpreted and understood.
Figure 1.
Survey branch points.
Participants and Recruitment Strategy
The study was conducted March–November 2016. Survey administration procedures were standardized across sites and managed by each site’s eConsult implementation team. The sampling process included the following procedures. All orders for an eConsult or standard referral generated during adult (≥ 18 years) primary care visits at the academic medical center and satellite clinics were identified monthly. To reduce potential bias from provider or clinic-level influences on patient experience, orders from PCPs who did not submit at least one eConsult in the previous month were removed. Referral orders to specialty departments not participating in eConsult were also removed in order to enable comparison of patients’ experiences with eConsult and referral in the same specialties. Additionally, all patients with more than one order in a given recruitment month were removed. This approach was deemed necessary after initial cognitive interviews revealed that the survey’s focus on one specific clinical encounter was confusing for patients with multiple recent clinical interactions and referrals and that this confusion could bias survey responses.
A web-based mode of survey administration was selected. Surveying patients by phone was not feasible given budget constraints, and a mailed version was ruled out because recent research shows that web-based surveys yield comparable results with mail more quickly and at lower costs.16, 17 Invitations with a link to the survey were distributed by email once a month, timed approximately 2–6 weeks after the primary care encounter. A reminder was sent to non-respondents 1 week after the initial invitation. The survey was administered at all nine sites using Qualtrics software (Provo, UT). All Qualtrics accounts and survey data were located on secure, HIPAA-compliant servers. The institutional review boards at all sites exempted the study from review.
Statistical Methods
Data were summarized overall, by site, and by eConsult or referral order using means and standard deviations for continuous variables and counts and percents for categorical and ordinal variables. Responses from patients recognizing their order type were compared using chi-squared statistics. To identify factors specific to referrals or eConsults, binary and multivariate logistic models were used to examine (a) whether demographic variables and experience with PCP predicted recognition of eConsult/referral service and satisfaction with service and (b) whether satisfaction with service and experience with PCP predicted preference for referral or eConsult for future health problems. Patients’ experiences with the PCP visit that led to the referral or eConsult were captured in a series of five questions that assessed trust and confidence in the PCP and satisfaction with the visit. These questions were aggregated to form a composite measure that we refer to here as “PCP trust” (see online Appendix A, Questions 4–8). Complete case analysis with listwise deletion was used for modeling, with models controlling for cross-site differences in PCP trust and patient demographics (specifically ethnicity, health status, age group, education group). All other analyses excluded missing responses (pairwise deletion). All analyses used P < 0.05 as the criterion for statistical significance and were analyzed using SPSS Statistics V25.18
RESULTS
A total of 29,291 email invitations were sent (3515 [12%] eConsult; 25,776 [88%] referral) and 8087 responses were received. The response rate was 26.7% (N = 938) for eConsult and 27.7% (N = 7148) for referral patients, with an overall response rate of 27.6%. The mean age of respondents was 57.5 and 64.5% were women. Self-reported education level showed 61.5% reporting a 4-year college degree or higher and 88.4% reporting some college or higher. Nearly half of respondents reported very good or excellent health and 84% of the study population identified as white. Statistically significant differences in demographic characteristics between eConsult and traditional referral patients included age and education. The average age for respondents who received an eConsult (58.5 [SD = 16.4]) was higher than that for those who received a referral (57.3 [SD = 15.7]), P = 0.040, while the proportion of patients with a 4-year college degree or higher was 58.4% for eConsult and 61.9% for referral (P = 0.042 chi-square test). (See Table 1 for full demographic details.)
Table 1.
Participant Characteristics
| Demographic category and total number of respondents* | eConsult (n = 938) | Referral (n = 7149) | |
|---|---|---|---|
| Women (938 eConsult/7149 referral) | 594 (63%) | 4619 (65%) | |
| Age (937 eConsult/7140 referral) | 18–24 | 21 (2%) | 159 (2%) |
| 25–34 | 79 (8%) | 662 (9%) | |
| 35–44 | 97 (10%) | 731 (10%) | |
| 45–54 | 145 (15%) | 1159 (16%) | |
| 55–64 | 212 (23%) | 1799 (25%) | |
| 65–74 | 229 (24%) | 1768 (25%) | |
| 75+ | 154 (16%) | 862 (12%) | |
| Education (884 eConsult/6658 referral) | 8th grade or less | 5 (< 1%) | 33 (< 1%) |
| Some high school, but did not complete | 19 (2%) | 84 (1%) | |
| High school graduate or GED | 114 (13%) | 622 (9%) | |
| Some college or 2-year degree | 230 (26%) | 1797 (27%) | |
| 4-year college degree | 204 (23%) | 1615 (24%) | |
| More than 4-year college degree | 312 (35%) | 2507 (38%) | |
| Race (879 eConsult/6484 referral) | White | 739 (85%) | 5441 (84%) |
| Black/African American | 18 (2%) | 180 (3%) | |
| Asian | 59 (7%) | 477 (7%) | |
| Native Hawaiian and other Pacific Islander | 46 (5%) | 334 (5%) | |
| American Indian or Alaska Native | 1 (< 1%) | 11 (< 1%) | |
| Other or multiple selections | 7 (< 1%) | 35 (< 1%) | |
| Ethnicity† (867 eConsult/6540 referral) | Hispanic/Latino | 51 (6%) | 417 (6%) |
| Overall Health (884 eConsult/6688 referral) | Excellent | 123 (14%) | 876 (13%) |
| Very good | 273 (31%) | 2310 (35%) | |
| Good | 308 (35%) | 2258 (34%) | |
| Fair | 142 (16%) | 1016 (15%) | |
| Poor | 38 (4%) | 219 (3%) | |
*Not all questions received responses; percentages were computed using the total number of respondents indicated in the first column. Multivariate models used complete case analysis
†Race and ethnicity were asked separately following current Office of Management and Budget (OMB) standards
Order Recognition
When asked if they had an in-person visit, phone call, or electronic message exchange with the identified primary care provider in the prior 6 weeks, the majority of respondents (86.9%; N = 7030) answered yes. Among those who recalled their primary care interaction, 64.9% (N = 4561) correctly identified that the PCP planned to get an advice from another physician or referred them for an office visit with a specialist, and correct identification for referrals was higher 67.2% (4174) compared with eConsults 47.1% (387). Across sites, rates of correct recognition ranged from 33 to 55% for eConsult and 60 to 78% for referral (online Appendix B).
A multivariable regression model using complete cases (eConsult N = 670; referral N = 5121) showed that respondents reporting very good or excellent health were significantly more likely than those with poorer health to recognize their PCP’s intention to use an eConsult (OR 1.62, 95% CI 1.18–2.23; P = 0.003) or to place a referral (OR 1.16, 95% CI 1.03–1.31; P = 0.019). Educational attainment was similarly associated with consult type recognition, with respondents with a 4-year college degree or higher more likely to recognize their PCP’s eConsult (OR 1.40, 95% CI 1.02–1.94; P = 0.040) or referral (OR 1.39, 95% CI 1.23–1.58; P < 0.001) decision. All results reported below are limited to respondents who identified their correct order type.
Comparing Patients’ Experiences Along eConsult and Referral Trajectories
Nearly all patients who recognized that their PCP had sought advice from a specialist through an eConsult or traditional referral agreed with their PCP’s decision to seek specialist input (eConsult N = 370 [98.9%]; referral N = 4115 [99.1%]). In terms of timeliness of care, 97.1% (N = 303) of eConsult patients reported receiving information about the specialist’s advice by the time of survey completion, whereas only 55.8% (N = 2064) of referral patients had completed an office visit with the specialist.
Among the 87.4% (N = 3625) of traditional referral patients who had scheduled a specialist appointment before responding to the survey, most (77.8%; N = 2797) reported being able to obtain the appointment as soon as needed. A higher percentage of eConsult patients agreed that communication about the specialist’s advice (95.8%; N = 297) was prompt. Figure 2 details how specialists’ eConsult advice was communicated to patients.]-->
Figure 2.
How eConsult recommendations were conveyed to patients.
When asked about overall satisfaction with the specialist’s recommendations, 81.4% (N = 249) of eConsult and 81.9% (N = 1675) of referral patients reported that they were satisfied or very satisfied. The specialist’s recommendations were explained clearly according to 97.1% (N = 301) of eConsult and 95.2% (N = 1952) of referral patients, while 80.4% (N = 250) of eConsult and 93.6% (N = 1913) of referral patients reported having had an opportunity to ask questions about the specialist’s recommendations (Fig. 3).]-->
Figure 3.
Comparison of eConsult and referral patients’ experience receiving specialist advice.
In multivariable regression models using complete cases, and adjusting for sample size and demographic differences across sites, we found that patients with high PCP trust (using the 5-question composite measure) were more likely to be satisfied with the specialist’s recommendations (eConsult OR 10.63, 95% CI 2.95–38.32; P < 0.001; N = 251; referral OR 2.86, 95% CI 1.85–4.42; P < 0.001; N = 1771). No other patient factors were associated with satisfaction, nor did we find significant cross-site differences in patient satisfaction.
Preferences for a Similar Problem in the Future
When asked to choose between eConsult or traditional referral for a similar problem in the future, 78.1% (N = 289) of eConsult and 67.6% (N = 2664) of referral patients chose the same service that they had received. Nearly one-third (32.4%; N = 1278) of referral patients said that they would prefer an eConsult in the future, whereas 21.9%; (N = 81) of eConsult patients would opt for traditional referral in the future (Fig. 4).]-->
Figure 4.
Preference for management of a similar problem in the future.
Multivariable regression analyses with complete cases were used to better understand factors associated with a preference for an eConsult in the future. Patients with a referral (N = 3462) were less likely to prefer eConsult in the future if they self-reported as white and not Hispanic (OR 0.65, 95% CI 0.55–0.78; P < 0.001), in very good or excellent health (OR 0.83, 95% CI 0.71–0.96; P = 0.011), or having completed a college degree or higher (OR 0.64, 95% CI 0.55–0.75; P < 0.001). Demographic variables were not predictive of order preference among eConsult patients. Patients who reported higher trust in and satisfaction with the PCP (PCP trust measure) were more likely to prefer eConsult in the future (OR 1.38, 95% CI 1.02–1.86; P = 0.035; N = 3770). When this same analysis was stratified by order type, this finding held true for those who had an eConsult (OR 6.74, 95% CI 2.49–18.27; P < 0.001; N = 308) but was not significant for those who had been referred (OR 1.22, 95% CI 0.89–1.67; P = 0.228; N = 3462).
DISCUSSION
Our results show that patients whose PCP placed an eConsult order were highly satisfied with the experience, to a similar degree as those who had already completed a specialty visit after a referral. Nearly all eConsult and referral patients agreed with the PCP’s decision to consult with a specialist. A large majority of eConsult patients reported receiving information about the specialist’s response in a timely manner. This contrasts with patients who had a referral, among whom nearly half had not yet completed an office visit with a specialist.
Interestingly, we found that a good primary care experience was predictive of higher satisfaction with the specialist’s recommendations among both groups of respondents. These findings suggest that primary care encounters and relationships can have an impact on patients’ experiences with clinicians and clinical services in other settings.
Nearly one-third of patients who were referred to see a specialist in person, and nearly four out of five patients whose PCP used the eConsult service, expressed a preference for eConsult for a similar problem in the future. Given that the eConsult service had been recently introduced at participating medical centers, we interpret this finding to mean that eConsult is an acceptable means of obtaining specialist input not only for the majority of patients who received the service but also for many patients with no prior knowledge of the service. Patients who had a good experience with their PCP were more likely to want an eConsult in the future, suggesting that confidence in one’s PCP increases the acceptability of this novel service.
Although we found statistical differences in age and education between eConsult and referral groups, the differences were quite small and not deemed to be clinically meaningful; regardless, regression analyses controlled for the impact of these characteristics. We found that referral patients who identified as white and non-Hispanic, had a college degree or higher, or were in better health were less likely to prefer an eConsult for a similar problem in the future. These findings cannot be explained without further research into the reasons for patients’ preferences, but we surmise that race/ethnicity and education may be serving as proxies for socioeconomic status and a bias toward in-person specialty care among those with more privilege.
It is concerning that nearly over half of eConsult patients, and nearly a third of referral patients, were not aware of their PCP’s intention to place an eConsult or referral order. The low rate of recognition among eConsult patients may be explained in several ways. First, PCPs may not always tell patients they are using eConsult and may decide in some cases to discuss the specialist’s advice only in the event that it prompts a change in management of the patient’s condition. Second, patients may not consider a virtual consultation between clinicians to be a distinct or noteworthy service, particularly when the patient is not being billed for the service (as was the case here).
Inaccurate recall among referral patients suggests that communication gaps can occur early in the referral process. Research has demonstrated high rates of incomplete referrals,19 whereby patients do not receive the specialty care intended by the referring provider. For instance, a recent study at a large academic medical center showed that only 35% of referrals from PCPs to specialists resulted in a completed appointment.20 Our study demonstrates that one source of incomplete referrals may be a lack of adequate understanding of the referral from the outset. Moreover, many patients have complex health problems and are being cared for by multiple health care providers. Although our study was designed to reduce confusion by limiting inclusion to patients with only one eConsult or referral order in the previous month, it is possible that survey respondents had difficulty recollecting the specific details of the encounter in question.
Our study has several limitations. Use of a web-based survey prevented us from recruiting patients without an email address, which may have biased our sample toward respondents with higher digital literacy and educational attainment. Our response rate was relatively low, which might also have introduced nonresponse bias. Also, the study population was less ethnically and racially diverse than the overall US population, so we were unable to adequately capture the experiences of underrepresented minorities and underserved communities. Research on underserved patients’ views about telemedicine suggests that there are fears and concerns about remote consultation as well as perceived advantages and that perspectives vary considerably across racial/ethnic groups.21 In studies of eConsult specifically, patients have reported concerns about access to specialty care.22 We hope that the relationship between care preferences, socioeconomic status, and access to health services is explored in future evaluations of eConsult acceptability among patients.
Another limitation was introduced by excluding patients with more than one order in the previous month. This limits the generalizability of our findings, particularly because patients with multiple orders may be more positively inclined toward a service that reduces wait times for specialist advice. Indeed, our focus group study with patients at the same academic medical centers found that medically complex patients with multiple providers were particularly enthusiastic about eConsult.23 Item nonresponse bias may have been introduced by eliminating missing responses from descriptive analyses. Moreover, missing data may have biased multivariate model results, although a comparison of participant characteristics in Table 1 with those of participants used for modeling (complete case analysis) did not demonstrate notable demographic differences. Finally, the survey was offered exclusively in English, which limits our insights into the experiences and preferences of non-English-speaking patients.
CONCLUSIONS
Patients’ experiences with eConsult have not been as closely or widely examined as those of clinicians. Our results help to fill this gap by demonstrating that a high proportion of patients find eConsult to be an acceptable strategy for the management of their medical condition and that trust and confidence in one’s PCPs are crucial ingredients for creating a satisfying eConsult experience.
There are prima facie concerns that patients might perceive services like eConsult to be a form of rationing or a lower quality of clinical care. By comparing eConsult and traditional referral perceptions and experiences, this study provides evidence that appears to counter these concerns. Nonetheless, the lack of awareness of eConsult among many patients who were beneficiaries of the service warrants an increased effort to include them in eConsult decision-making and communication. As more payers elect to cover eConsult services (including CMS, which began coverage in 2019), the question of patient awareness takes on additional importance. A charge for a co-payment, for instance, may cause confusion for patients who do not recognize having received the service and may reduce eConsult acceptability.23
Electronic Supplementary Material
(DOCX 73 kb)
Acknowledgments
We offer enthusiastic thanks to the many patients who took time to share their perspectives with us. We are also enormously grateful to all sites’ implementation teams for their diligent efforts to administer the survey and harmonize results. Finally, we owe a special debt of gratitude to Tristan Grogan for his expert and timely contributions to data analysis.
Funding Information
The study reported here was made possible by Grant Number 1C1CMS331324 from the Department of Health and Human Services, Center for Medicare & Medicaid Services, and by the University of California Center for Health Quality and Innovation.
Compliance with Ethical Standards
The institutional review boards at all sites exempted the study from review.
Conflict of Interest
The authors declare that they do not have conflicts of interest.
Disclaimer
The contents of this publication are solely the responsibility of the authors and do not necessarily represent the official views of the US Department of Health and Human Services or any of its agencies.
Footnotes
Prior presentations
Preliminary results from this study were presented as a poster at the SGIM Annual Meeting in April, 2017.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Lai L, Liddy C, Keely E, et al. The impact of electronic consultation on a Canadian tertiary care pediatric specialty referral system: A prospective single-center observational study. Jhaveri R, ed. PLoS ONE. 2018;13(1):e0190247. [DOI] [PMC free article] [PubMed]
- 2.Barnett ML, Yee HF, Mehrotra A, Giboney P. Los Angeles Safety-Net Program eConsult System Was Rapidly Adopted and Decreased Wait Times to See Specialists. Health Aff (Millwood). 2017;36(3):492–499. doi: 10.1377/hlthaff.2016.1283. [DOI] [PubMed] [Google Scholar]
- 3.Gleason N, Prasad PA, Ackerman S, et al. Adoption and impact of an eConsult system in a fee-for-service setting. Healthcare. 2017;5(1–2):40–45. doi: 10.1016/j.hjdsi.2016.05.005. [DOI] [PubMed] [Google Scholar]
- 4.Chen AH, Kushel MB, Grumbach K, Yee HF. A Safety-Net System Gains Efficiencies Through “eReferrals” To Specialists. Health Aff (Millwood). 2010;29(5):969–971. doi: 10.1377/hlthaff.2010.0027. [DOI] [PubMed] [Google Scholar]
- 5.Liddy C, Hogel M, Blazkho V, Keely E. The current state of electronic consultation and electronic referral systems in Canada: an environmental scan. Stud Health Technol Inform. 2015;209:75–83. [PubMed] [Google Scholar]
- 6.Vimalananda VG, Gupte G, Seraj SM, et al. Electronic consultations (e-consults) to improve access to specialty care: A systematic review and narrative synthesis. J Telemed Telecare. 2015;21(6):323–330. doi: 10.1177/1357633X15582108. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Liddy C, Drosinis P, Keely E. Electronic consultation systems: worldwide prevalence and their impact on patient care-a systematic review. Fam Pract. 2016;33(3):274–285. doi: 10.1093/fampra/cmw024. [DOI] [PubMed] [Google Scholar]
- 8.Lee MS, Ray KN, Mehrotra A, Giboney P, Yee HF, Jr, Barnett ML. Primary Care Practitioners’ Perceptions of Electronic Consult Systems. JAMA Intern Med. 2018;178(6):782–789. doi: 10.1001/jamainternmed.2018.0738. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Kwok J, Olayiwola JN, Knox M, Murphy EJ, Tuot DS. Electronic consultation system demonstrates educational benefit for primary care providers. J Telemed Telecare. 2018;24(7):465–472. doi: 10.1177/1357633X17711822. [DOI] [PubMed] [Google Scholar]
- 10.Liddy C, Afkham A, Drosinis P, Joschko J, Keely E. Impact of and Satisfaction with a New eConsult Service: A Mixed Methods Study of Primary Care Providers. J Am Board Fam Med. 2015;28(3):394–403. doi: 10.3122/jabfm.2015.03.140255. [DOI] [PubMed] [Google Scholar]
- 11.Keely E, Drosinis P, Afkham A, Liddy C. Perspectives of Champlain BASE Specialist Physicians: Their Motivation, Experiences and Recommendations for Providing eConsultations to Primary Care Providers. Stud Health Technol Inform. 2015;209:38–45. [PubMed] [Google Scholar]
- 12.Deeds S, Dowdell K, Chew L, Ackerman S. Implementing an Opt-in EConsult Program at Seven Academic Medical Centers. JGIM. 2019;34(8):1427–1433. doi: 10.1007/s11606-019-05067-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Rodriguez KL, Burkitt KH, Bayliss NK, et al. Veteran, primary care provider, and specialist satisfaction with electronic consultation. JMIR Med Inform. 2015;3(1):e5. doi: 10.2196/medinform.3725. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Palen TE, Price D, Shetterly S, Wallace KB. Comparing virtual consults to traditional consults using an electronic health record: an observational case-control study. BMC Med Inform Decis Mak. 2012;12:65. doi: 10.1186/1472-6947-12-65. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.CAHPS Clinician & Group Survey. https://www.ahrq.gov/cahps/surveys-guidance/cg/index.html. Accessed Dec. 23, 2019.
- 16.Bergeson SC, Gray J, Ehrmantraut LA, Laibson T, Hays RD. Comparing Web-based with Mail Survey Administration of the Consumer Assessment of Healthcare Providers and Systems (CAHPS®) Clinician and Group Survey. Prim Health Care. 2013;3:1000132. doi: 10.4172/2167-1079.1000132. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Bojcic JL, Sue VM, Huon TS, Maletis GB, Inacio MC. Comparison of paper and electronic surveys for measuring patient-reported outcomes after anterior cruciate ligament reconstruction. Perm J. 2014;18(3):22–26. doi: 10.7812/TPP/13-142. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.IBM Corp. Released 2017. IBM SPSS Statistics for Windows, Version 25.0. Armonk, NY: IBM Corp.
- 19.Weiner, Perkins, Callahan Errors in Completion of Referrals among Older Urban Adults in Ambulatory Care. J Eval Clin Pract. 2010;16(1):76–81. doi: 10.1111/j.1365-2753.2008.01117.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Patel MP, Schettini P, O’Leary CP, Bosworth HB, Anderson JB, Shah KP. Closing the Referral Loop: an Analysis of Primary Care Referrals to Specialists in a Large Health System. J Gen Intern Med. 2018;33(5):715–721. doi: 10.1007/s11606-018-4392-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.George S, Hamilton A, Baker RS. How Do Low-Income Urban African Americans and Latinos Feel about Telemedicine? A Diffusion of Innovation Analysis. Int J Telemed Appl. 2012;2012:715194. doi: 10.1155/2012/715194. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Joschko J, Liddy C, Moroz I, Reiche M, Crowe L, Afkham A, Keely E. Just a click away: exploring patients’ perspectives on receiving care through the Champlain BASE™ eConsult service. Fam Pract. 2018;35(1):93–98. doi: 10.1093/fampra/cmx073. [DOI] [PubMed] [Google Scholar]
- 23.Ackerman S, Dowdell K, Clebak K, Quinn M, Shipman S. Patients weigh in on eConsult: A qualitative study of communication and decision-making preferences at five U.S. academic medical centers. Ann Fam Med. 2020;18(1):35–41. doi: 10.1370/afm.2487. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
(DOCX 73 kb)