Abstract
PURPOSE
Communication between specialists and primary care clinicians is suboptimal, and access to referrals is often limited, which can lead to lower quality, inefficiency, and errors. An electronic consultation (e-consultation) is an asynchronous, non-face-to-face consultation between a primary care clinician and a specialist using a secure electronic communication platform. The purpose of this study was to conduct a randomized controlled trial of e-consultations to test its efficacy and effectiveness in reducing wait times and improving access to specialty care.
METHODS
Primary care clinicians were randomized into a control (9 traditional) or an intervention (17 e-consultation) arm for referrals to cardiologists. Primary care clinicians were recruited from 12 practice sites in a community health center in Connecticut with mainly medically underserved patients. Two end points were analyzed with a Cox proportional hazards model where the hazard of either a visit or an e-consultation was linked to study arm, sex, race, and age.
RESULTS
Thirty-six primary care clinicians participated in the study, referring 590 patients. In total, 69% of e-consultations were resolved without a visit to a cardiologist. After adjusting for covariates, median days to a review for an electronic consultation vs a visit for control patients were 5 and 24, respectively. A review of 6-month follow-up data found fewer cardiac-related emergency department visits for the intervention group.
CONCLUSION
E-consultation referrals improved access to and timeliness of care for an underserved population, reduced overall specialty utilization, and streamlined specialty referrals without any increase in adverse cardiovascular outcomes. e-consultations are a potential solution for improving access to specialty care.
INTRODUCTION
The number of ambulatory care visits that result in a referral to another health care clinician has doubled during the past decade.1 Timely access and good communication between clinicians are essential for quality, efficiency, and patient safety. In a national survey, however, only 34% of specialists reported routinely receiving information from referring primary care clinicians, and only 62% of the clinicians reported reliably receiving information back from the specialist.2 This suboptimal exchange of information leads to an increase in medical errors, wasteful spending, and poor quality of care.3,4
In addition, access to subspecialty care is often limited, especially for medically underserved populations. At least 1 in 4 medical encounters at community health centers result in a referral to a specialist.5 Obtaining appointments for these referrals is challenging because so few specialists are willing to accommodate them.5,6 This imbalance in supply and demand leads to waiting times for appointments that can be as long as 1 year.7–9 Inadequate access to care is a major contributor to health care disparities, higher rates of disability, chronic disease sequelae, and death.5,10
Electronic consultations (e-consultations) are an emerging technology that may help address these problems by improving communication and information exchange between health care clinicians and providing timely access to efficient and high-quality care for patients.11,12 An e-consultation is an asynchronous, non-face-to-face consultation between a primary care clinician and a specialist using a secure electronic communication platform. In many cases, the clinician can manage the patient’s condition with guidance from the specialist provided via the e-consultation. In other cases, additional workup or an in-person visit is recommended.
Observational studies suggest the potential for e-consultations to reduce not only waiting times for specialty care13 but also the need for face-to-face consultations from 8.9% to 51% depending on setting and specialty.7–10,13–15 There are no published studies, however, showing the effect of an e-consultation program on utilization and clinical outcomes. In addition, most e-consultation programs have been implemented in integrated delivery systems with a shared electronic health record (EHR) or a designated specialty care network.10,16–18
To address these gaps in the evidence, we conducted a cluster-randomized controlled trial of cardiologists’ e-consultations in a large and geographically dispersed, multisite federally qualified health center. We chose cardiology as the specialty because it had high rates of referral and limited access.
METHODS
Design Overview
The cardiology e-consultation study is a prospective, cluster-randomized, controlled intervention trial carried out between October 2011 and December 2013 at Community Health Center, Inc (CHCI) in Connecticut.
Setting and Participants
CHCI is a level-3 recognized patient-centered medical home providing comprehensive primary medical, behavioral, and dental care to medically underserved patients. More than 60% of CHCI patients are racial/ethnic minorities, more than 90% are at or below the 200% federal poverty level, more than 60% have state Medicaid insurance, and almost 25% are uninsured. Patients receive primary medical care by internists, family physicians, pediatricians, nurse practitioners, and physician assistants. Most of CHCI’s practice sites rely on hospitals and specialists in their own communities for referrals, or they send their patients to the state-funded University of Connecticut Health Center (UCHC), located in the central part of the state.
All primary care clinicians caring for adult patients at CHCI’s 12 primary care centers who worked at least 30 hours per week were eligible and invited to participate in the study. E-consultations were provided by cardiologists from UCHC, for which cardiologists were reimbursed $25 each.
Randomization and Intervention
All participating primary care clinicians entered into the study voluntarily, signed a written consent form, and were informed that they could discontinue participation at any time. All patients referred to a cardiologist by the clinician participating in the study were included in the study. Enrolled clinicians were block-randomized into the intervention arm (e-consultation referral) or the control arm (traditional referral) of the study using fixed-size blocks of 4; investigators were blinded to the block size, block number, and sequence in the block. One author who was not involved in data collection and final data analysis generated the allocation sequence and assigned participants to the study groups. Of the 2 principal investigators, 1 was blinded to clinician participation and group assignment, whereas the other provided oversight of clinical research activities. Because of the nature of the intervention, participants were not blinded. Research staff who were abstracting medical records were blinded to the patient study group. The CHCI Institutional Review Board approved this protocol, including a waiver of patient informed consent. The trial was registered through Clinicaltrials.gov (NCT02376855).
The intervention consisted of an e-consultation pathway and a standardized protocol for clinicians to obtain cardiology consultations using a secure peer-to-peer messaging module embedded within the EHR. The intervention period lasted 1 year, ending on July 31, 2013. During this period, all adult cardiology referrals from intervention clinicians were sent via the e-consultation pathway unless they were deemed urgent by the clinician or were for patients who had an already established relationship with a cardiologist.
Intervention clinicians were trained to use the e-consultation module to create referrals in the same way they created traditional referrals: state the reason(s) for the referral, include any relevant test results, records or reports; and submit through the EHR. The e-consultation referrals were then sent electronically to the cardiology consultants by a referral coordinator. A team of 3 cardiologists from UCHC maintained a coverage schedule and responded to e-consultation referrals within 2 business days.
Responses from the cardiologist were received by the referring clinician as an incoming message within the EHR. Responses were case-specific and generally contained recommendations for management, additional testing, or a face-to-face cardiologist’s visit. The clinician was responsible for considering and acting upon recommendations and determining when an e-consultation was complete. When a face-to-face visit was recommended, the clinician referred the patient to a cardiologist based on location and the patient’s preference. Notes from the e-consultation accompanied the referral to any cardiologist. Clinicians randomized to the control group continued to refer patients using the standard CHCI referral process, which also involved the creation and submission of a referral in the EHR. Referral coordinators then processed the referral, contacted specialist offices to obtain an appointment, and faxed relevant clinical information and attachments. Consultation notes from the cardiologist were returned via fax and scanned into the EHR.
Outcomes and Follow-up
The primary study outcome was the time to a consultation with a cardiologist. This outcome was defined as either a completed e-consultation that did not require further follow-up, or a documented face-to-face visit with a cardiologist. Secondary outcomes included the completion rate of referrals to cardiologists and the number of face-to-face visits in each group. Other secondary outcomes included adverse events, including death, myocardial infarction, coronary artery bypass grafting, urgent or emergent cardiac catheterization and/or angioplasty, hospital utilization for potential cardiac complaints or events, and emergency department utilization for potential cardiac complaints or events. All adverse events were measured as binary outcomes.
Demographic, clinical, and operational data, including the outcome of the consultation request, was collected for all patients referred to a cardiologist by primary care clinicians in the study for the 12-month intervention period and for a 6-month follow-up period. Faxes and other documents contained in the charts of all patients were reviewed by trained research staff using a formal chart abstraction tool to identify any adverse events, tests, and results, including additional specialty visits, diagnostic procedures, hospitalizations, and emergency department visits. One principal investigator (D.A.) conducted a secondary review of any case for which there were questions. In addition, we reviewed any question contained in each consultation. The clinician’s experience was assessed with a brief online survey administered to the intervention group at baseline, mid-intervention, and 12 months. The survey assessed general satisfaction with the intervention, ease of use, and impact on workload, and these findings will be reported separately.
Statistical Analysis
Primary analysis involved analyzing the event time end point with a Cox proportional hazards model,19 where the hazard (instantaneous risk) of either a face-to-face visit or completed e-consultation was linked to study arm, sex, race, and age using the R package “survival.”20 In preliminary analysis, a Cox model with time-varying coefficient21 was fitted to test the proportional hazards assumption using the R package “timereg.”22 The null hypothesis of a constant coefficient for the treatment group over time could not be rejected, indicating that the proportional hazards assumption was appropriate. Because the unit of randomization was the clinician, we fitted the proportional hazards model using the coxph function, which allowed us to specify a marginal model that adjusted for the clustering of patients within clinicians.23,24 Secondary analysis included differences in adverse events and emergency departments visits, as well as hospitalizations between study arms.
RESULTS
Forty-four primary care clinicians were invited, and 36 gave consent and were randomized. Nineteen clinicians were assigned to the control group, and 17 to the e-consultation group. The demographic and baseline clinical characteristics of the participating clinicians and their 590 patients referred during the study period are displayed in Table 1. Clinicians’ mean age and years in practice were 39 and 8 years, respectively; most were family physicians. Mean age of patients referred to a cardiologist by study clinicians was 53 years; 39% were white and 35% Hispanic or Latino; 54% were female. Three clinicians dropped out of the study when they left the organization; the remaining clinicians stayed in the study for the duration of the trial. Patients in each study arm had very similar demographic profiles. In addition, patients referred by clinicians in each study arm were well matched with regard to baseline clinical characteristics and cardiac risk as assessed by diabetes or hypertension status, smoking, or Framingham cardiovascular risk scores.
Table 1.
Demographic Characteristics of Primary Care Clinicians and Patients
| Characteristic | Intervention | Control |
|---|---|---|
| Clinician, No. | 17 | 19 |
| Age, mean (SD), y | 37.3 (7.5) | 40.5 (10.1) |
| Years in practice, mean (SD) | 6.1 (7.2) | 10.1 (9.6) |
| Female, No. (%) | 13 (76) | 12 (63) |
| Race | ||
| Asian, No. (%) | 3 (18) | 5 (26) |
| Black, No. (%) | 3 (18) | 2 (11) |
| Hispanic, No. (%) | 0 | 1 (5) |
| White, No. (%) | 11 (65) | 11 (58) |
| Clinician specialty | ||
| Family medicine physician, No. (%) | 8 (47) | 13 (68) |
| Internal medicine physician, No. (%) | 3 (18) | 1 (5) |
| Nurse practitioner/physician assistant, No. (%) | 6 (35) | 5 (26) |
| Patient, No. | 229 | 361 |
| Age, mean (SD), y | 51.9 (15.8) | 53.8 (13.3) |
| Female, No. (%) | 119 (52) | 202 (56) |
| Race | ||
| White, No. (%) | 94 (41) | 134 (37) |
| Black, No. (%) | 39 (17) | 53 (15) |
| Hispanic, No. (%) | 68 (30) | 140 (39) |
| Other, No. (%) | 28 (12) | 34 (9) |
| Smoking status | ||
| Current every day smoker, No. (%) | 58 (25) | 93 (26) |
| Current some day smoker, No. (%) | 2 (0.9) | 1 (0.2) |
| Former smoker, No. (%) | 44 (19) | 94 (26) |
| Never smoker, No. (%) | 94 (41) | 144 (40) |
| Smoker, current status unknown, No. (%) | 30 (13) | 29 (8) |
| Body mass index, mean (SD) | 30.8 (8.4) | 31.6 (7.8) |
| Total cholesterol, mean (SD), mg/dL | 192.5 (50.7) | 188.4 (46.0) |
| Diagnosis of diabetes, No. (%) | 63 (28) | 104 (29) |
| Framingham risk score, mean (SD) | 13.9 (10.4) | 14.0 (10.1) |
| Insurance status | ||
| Medicaid, No. (%) | 143 (62) | 217 (60) |
| Medicare, No. (%) | 32 (14) | 65 (18) |
| Other public, No. (%) | 1 (0) | 0 |
| Private, No. (%) | 21 (9) | 47 (13) |
| Uninsured, No. (%) | 32 (14) | 32 (9) |
| Treatment priority | ||
| Routine, No. (%) | 205 (90) | 291 (81) |
| Urgent, No. (%) | 22 (10) | 67 (19) |
| Critical, No. (%) | 2 (1) | 3 (1) |
Referral pathways and outcomes are displayed in Figure 1. In the intervention group, 120 of 229 cardiology referrals (52%) were sent electronically as an e-consultation. The remaining 109 referrals (48%) were not sent through e-consultation pathway because of urgency (23 referrals) or the existence of an established relationship between the patient and a local cardiologist (86 referrals). The percentage of clinicians’ referrals sent via an e-consultation in the intervention arm varied somewhat. Of the 12 clinicians who made 5 or more cardiology referrals during the intervention, 8 sent 50% or more via the e-consultation pathway, with 17 of 19 clinicians sending 89%. Four clinicians sent fewer than 50% of their referrals through the e-consultation pathway. The clinician with the lowest percentage of e-consultations sent none of 9 referrals through the e-consultation pathway. The most common reasons for consultation referrals made through the e-consultation pathway were abnormal electrocardiogram (ECG) findings, after which came chest pain, preoperative assessment, and hypertension. For the patients referred directly for face-to-face consultations, the most common reason was follow-up for established coronary artery disease, then chest pain, heart failure, hypertension, and abnormal findings on an ECG.
Figure 1.
Workflow and volume of cardiology referrals, August 1, 2012, through July 31, 2013.
Note: flow chart illustrates results of every referral during the 1-year study.
F2F = face-to-face appointment; e-consultation = electronic consultation.
aPatient died of noncardiac-related event.
For the 120 patients in the intervention group referred to a cardiologist via the e-consultation pathway, 83 (69.2%) were believed by the reviewing cardiologist to be manageable by the clinicians and not require a face-to-face visit. In 33 (27.5%) of the e-consultation referrals, the reviewing cardiologist recommended a face-to-face visit. The remaining 4 (3.3%) e-consultation referrals were never received or reviewed by the specialist reviewer because of technical problems with the e-consultation system. Patients with heart failure and follow-up of an established diagnosis of coronary artery disease were more commonly referred for a face-to-face visit, whereas questions about abnormal ECG findings were managed electronically.
Most e-consultation referrals were completed with 1 exchange (clinician-cardiologist-clinician). There was no statistically significant change in rates of referral during the course of the intervention in either study arm.
Table 2 presents results from Cox regression model assessing the impact of an e-consultation on the time between the request and completion of a consultation. These results reflect differences in the likelihood that a primary care referral will result in a completed consultation with a cardiologist, which for e-consultation patients was the date on which the cardiologist’s response occurred. The model presents an intention-to-treat analysis, as patients assigned to the intervention group who bypassed the e-consultation pathway were analyzed with the intervention group. Control variables included in the model were patients’ age, sex, and race and ethnicity (black, Hispanic, other race vs white), and priority status of the consultation request (routine vs urgent or critical).
Table 2.
Cox Regression Model Comparing Time to Cardiologist Consult
| Variable | Exponentiated Coefficient | 95% CI | P Value |
|---|---|---|---|
| Intervention | 1.45 | 1.14–1.76 | .019 |
| Control (omitted) | … | … | … |
| Black | 1.07 | 0.84–1.30 | .57 |
| Hispanic | 0.91 | 0.67–1.14 | .42 |
| Other race | 0.91 | 0.54–1.29 | .63 |
| White (omitted) | … | … | … |
| Age | 1.00 | 0.99–1.01 | .83 |
| Male | 0.88 | 0.71–1.06 | .16 |
| Female (omitted) | … | … | … |
| Routine priority | 0.84 | 0.50–1.19 | .33 |
| Urgent priority (omitted) | … | … | … |
Note: Model predicts time to date a consultation is provided by cardiologist where the date is either the date of e-consultation or date of face-to-face visit for those bypassing e-consultation pathway.
Results from this analysis indicate that patients in the intervention group were significantly more likely to receive a consultation from a cardiologist than were control patients. The exponentiated coefficient indicates that intervention patients were almost 1.5 times more likely to receive a consultation from a cardiologist than were control patients. None of the control variables included in the model were significant predictors of obtaining a consultation. Further analysis separating patients receiving e-consultations from patients assigned to the intervention group who bypassed the e-consultation pathway (because of a previous relationship with a cardiologist or being deemed an urgent case) indicated that those receiving e-consultations were almost 4 times (OR = 3.74, 95% CI, 2.60–5.39) more likely than control patients to have had their case reviewed by a cardiologist among those in the intervention group (data not shown).
To provide additional context for evaluating these results, Table 3 presents the median times to a cardiology visit and the percentage of patients not receiving a consultation with a cardiologist within 31 and 180 days among 3 groups: (1) those receiving an e-consultation referral, (2) those in the intervention group who received a traditional referral (ie, bypassed the e-consultation pathway), and (3) the control patients. Medians and percentages were derived from the Cox regression results presented in Table 2, which calculated time to occurrence of the e-consultation review or the face-to-face follow-up among patients receiving a traditional referral. Dramatic differences in the time to a consultation with a cardiologist were observed among patients in the 3 groups. The median number of days to a consultation for e-consultation patients was 5, compared with 29 and 24 among intervention patients receiving traditional referrals (ie, those that bypassed the e-consultation pathway) and control patients, respectively. The percentage of patients who did not receive a consultation with a cardiologist within 1 and 6 months of the initial referral also differed dramatically between groups.
Table 3.
Comparing Time to Consultation With Cardiologist Among Study Groups (N = 590 Referrals)
| Variable | Group
|
||
|---|---|---|---|
| Intervention, E-consultation | Intervention, Traditional | Control | |
| Median time to consultation, d | 5 | 29 | 24 |
| Patients not consulting cardiologist within 31 days of referral, % | 14 | 48 | 38 |
| Patients not consulting cardiologist within 180 days of referral, % | 6 | 21 | 16 |
Patients referred via the traditional consultation pathway had higher rates of emergency department utilization than those receiving e-consultation referrals during the 6-month follow-up (Table 4). Though the study was not powered to detect statistical differences, among the 21 emergency department visits for patients in the control group, 6 (29%) occurred in the time between the clinician’s request and the patient’s consultation with the cardiologist.
Table 4.
Six-month Follow-up of Cardiac Events Among Intervention and Control Patients
| Cardiac Event | Intervention
|
Control n = 361 | ||
|---|---|---|---|---|
| Total n = 229 | Traditional Pathway n = 109 | E-consultation Pathway n = 120 | ||
| Death from any cause, No. (%) | 0 (0) | 0 (0) | 0 (0) | 1 (0.3) |
| Death from cardiovascular causes, No. (%) | 0 (0) | 0 (0) | 0 (0) | 0 |
| Myocardial infarction, No. (%) | 0 (0) | 0 (0) | 0 (0) | 0 |
| Coronary artery bypass surgery, No. (%) | 0 (0) | 0 (0) | 0 (0) | 0 |
| Catheterization with stenting or angioplasty, No. (%) | 3 (1.3) | 1 (0.9) | 2 (1.7) | 2 (0.6) |
| Diagnostic catheterization, No. (%) | 1 (0.4) | 1 (0.9) | 0 | 6 (1.7) |
| Emergency department visits with possible cardiac symptoms, No. (%)a | 4 (1.7) | 3 (2.8) | 1 (0.8) | 21 (5.8) |
| Hospitalization for potential cardiac causes, No. (%) | 10 (4.4) | 7 (6.4) | 3 (2.5) | 19 (5.2) |
Includes chest pain, palpitations/arrhythmia, shortness of breath, and potential cardiac ischemia (rule out myocardial infarction).
Surveys assessing primary care clinicians’ experiences with an e-consultation were completed by 8 of 17 clinicians (47%) in the intervention group. Six clinicians (75%) believed that the e-consultation had either no impact or resulted in a decrease in their overall workload, requiring no more than 5 minutes to enter and send. All respondents were satisfied or highly satisfied with the quality and timeliness of the e-consultation responses.
DISCUSSION
The results from the first randomized study of referrals for an e-consultation reveal the potential to improve safely and efficiently the primary care–specialty consultation process using an electronic communication platform. Approximately one-half of all cardiology referrals met our criteria to be sent via e-consultation, and two-thirds of those did not require a face-to-face visit. Moreover, e-consultation reviews were completed, on average, almost a month sooner than those sent for a face-to-face consultation, even those deemed urgent by the referring clinician. Although there was some variability in use of the e-consultation pathway by individual clinicians, likely reflecting differences in case mix (more cases of urgent or established disease in some clinicians’ panels) or in a clinician’s comfort and assessment of the urgency of the requested consultation, most of the clinicians used that pathway for one-half or more referrals. Either way, these results show that a substantial number of consultations can be safely and more efficiently managed though a secure electronic exchange of information, with improved convenience for the patient as well. Careful evaluation of the impact of e-consultation on patient safety and quality of care was undertaken through detailed reviews of the clinical records, including hospital and emergency department documentation, in the 6 months after the intervention (Table 4). We did not find any evidence that patients managed by clinicians who requested an e-consultation had more adverse cardiac outcomes than patients of clinicians in the control group. These findings, coupled with the reduction in waiting time for appointments and need for face-to-face specialist consultation suggest that e-consultation referrals may improve efficiency without compromising the quality of care.
This study has several limitations. The study focused on a single specialty, and the impact of e-consultations could vary among specialties.10 In addition, analysis of the clinical outcomes of patients relied on chart review of faxed documents to determine rates of hospitalization and emergency department use, and it is possible that not all records were in the EHR. This problem, however, would affect findings equally in both groups. Studies using claims-level data may provide more accurate information. In addition, the study had a small sample size of primary care clinicians and patients, so broad conclusions cannot be drawn. Further research should be done to assess e-consultations in settings other than federally qualified health centers, though the benefits of waiting time reduction and equivalent quality are likely applicable to other settings. Patients’ experience was not assessed and is an area for future exploration. Finally, technical problems with the e-consultation system led to 3.3% of e-consultation requests not receiving a specialist review. The study was not designed to formally detect adverse events. Further studies on safety and adverse events, particularly exploring the potential challenges of technology, are needed.
We found that e-consultation referrals are effective at increasing access to specialty care for underserved populations, reducing need for face-to-face specialty care, streamlining specialty referrals, and potentially reducing emergency department utilization. These findings have critical policy implications as health systems seek new ways to improve quality and reduce cost. There is also growing recognition that the e-consultation pathway may be fundamental to the success of accountable care organizations as part of a comprehensive strategy to streamline costs, improve quality, and better coordinate care.22 Though further research is needed to recommend e-consultations for wide use across settings and specialties, e-consultations showed great promise in advancing integration of the patient-centered medical home into the larger medical neighborhood and potentially mitigating health disparities in access and treatment in our study population.
Acknowledgments
The authors would like to thank Ms Khushbu Khatri, Research Assistant, at the Community Health Center, Inc, Weitzman Institute, for her editorial assistance.
Footnotes
Conflicts of interest: authors report none.
Funding support: This work was supported by The Connecticut Health Foundation.
Disclaimer: The funder did not participate in the design, conduct, or analysis of this study, nor in the publication of the manuscript.
Previous presentations: Academy Health Annual Research Meeting, June 8–10, 2014, San Diego, California; National Association of Community Health Centers, FOM/IT Conference, October 28–29, 2014, Las Vegas, Nevada; American Telemedicine Association Conference, May 5–7, 2013, Austin, Texas.
References
- 1.Barnett K, McLachlan C, Hulbert J, Kassak K. Working together in rural South Dakota: integrating medical and chiropractic primary care. J Manipulative Physiol Ther. 1997;20(9):577–582. [PubMed] [Google Scholar]
- 2.O’Malley AS, Reschovsky JD. Referral and consultation communication between primary care and specialist physicians: finding common ground. Arch Intern Med. 2011;171(1):56–65. [DOI] [PubMed] [Google Scholar]
- 3.Bodenheimer T. Coordinating care—a perilous journey through the health care system. N Engl J Med. 2008;358(10):1064–1071. [DOI] [PubMed] [Google Scholar]
- 4.Mehrotra A, Forrest CB, Lin CY. Dropping the baton: specialty referrals in the United States. Milbank Q. 2011;89(1):39–68. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Cook NL, Hicks LS, O’Malley AJ, Keegan T, Guadagnoli E, Landon BE. Access to specialty care and medical services in community health centers. Health Aff (Millwood). 2007;26(5):1459–1468. [DOI] [PubMed] [Google Scholar]
- 6.Forrest CB, Shadmi E, Nutting PA, Starfield B. Specialty referral completion among primary care patients: results from the ASPN Referral Study. Ann Fam Med. 2007;5(4):361–367. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Chen AH, Kushel MB, Grumbach K, Yee HF. A safety-net system gains efficiencies through ‘eReferrals’ to specialists. Health Aff (Millwood). 2010;29(5):969–971. [DOI] [PubMed] [Google Scholar]
- 8.Kim Y, Chen AH, Keith E, Yee HF, Jr, Kushel MB. Not perfect, but better: primary care providers’ experiences with electronic referrals in a safety net health system. J Gen Intern Med. 2009;24(5):614–619. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Kim-Hwang JE, Chen AH, Bell DS, Guzman D, Yee HF, Jr, Kushel MB. Evaluating electronic referrals for specialty care at a public hospital. J Gen Intern Med. 2010;25(10):1123–1128. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Horner K, Wagner E, Tufano J. Electronic consultations between primary and specialty care clinicians: early insights. Issue Brief (Commonw Fund). 2011;23:1–14. [PubMed] [Google Scholar]
- 11.Wagner EH, Austin BT, Von Korff M. Organizing care for patients with chronic illness. Milbank Q. 1996;74(4):511–544. [PubMed] [Google Scholar]
- 12.Coleman K, Phillips K, Fund C. Providing Underserved Patients with Medical Homes: Assessing the Readiness of Safety-Net Health Centers. New York, NY: Commonwealth Fund; 2010. [PubMed] [Google Scholar]
- 13.Chen AH, Murphy EJ, Yee HF., Jr eReferral—a new model for integrated care. N Engl J Med. 2013;368(26):2450–2453. [DOI] [PubMed] [Google Scholar]
- 14.Scherpbier-de Haan ND, van Gelder VA, Van Weel C, Vervoort GM, Wetzels JF, de Grauw WJ. Initial implementation of a web-based consultation process for patients with chronic kidney disease. Ann Fam Med. 2013;11(2):151–156. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Palen TE, Price D, Shetterly S, Wallace KB. Comparing virtual consults to traditional consults using an electronic health record: an observational case-control study. BMC Med Inform Decis Mak. 2012;12(1):65. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Liddy C, Maranger J, Afkham A, Keely E. Ten steps to establishing an e-consultation service to improve access to specialist care. Telemed J E Health. 2013;19(12):982–990. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Liddy C, Rowan MS, Afkham A, Maranger J, Keely E. Building access to specialist care through e-consultation. Open Med. 2013;7(1):e1–e8. [PMC free article] [PubMed] [Google Scholar]
- 18.Keely E, Liddy C, Afkham A. Utilization, benefits, and impact of an e-consultation service across diverse specialties and primary care providers. Telemed J E Health. 2013;19(10):733–738. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Cox DR. Regression models and life tables (with discussion). J R Stat Soc [Ser B]. 1972;34(2):187–220. [Google Scholar]
- 20.Therneau T. A Package for Survival Analysis in S. version 2.38. http://CRAN.R-project.org/package=survival 2015.
- 21.Martinussen T, Scheike TH. Dynamic Regression Models for Survival Data (Statistics for Biology and Health). New York, NY: Springer; 2006. [Google Scholar]
- 22.Scheike TH, Zhang M-J. Analyzing competing risk data using the R timereg package. J Stat Softw. 2011;38(2). [PMC free article] [PubMed] [Google Scholar]
- 23.Wei L-J, Lin DY, Weissfeld L. Regression analysis of multivariate incomplete failure time data by modeling marginal distributions. J Am Stat Assoc. 1989;84(408):1065–1073. [Google Scholar]
- 24.Therneau TM, Grambsch PM. Modeling Survival Data: Extending the Cox Model. New York, NY: Springer; 2000. [Google Scholar]