Abstract
Background
Electronic referrals can improve access to subspecialty care in safety net settings. In January 2007, San Francisco General Hospital (SFGH) launched an electronic referral portal that incorporated subspecialist triage, iterative communication with referring providers, and existing electronic health record data to improve access to subspecialty care.
Objective
We surveyed primary care providers (PCPs) to assess the impact of electronic referrals on workflow and clinical care.
Design
We administered an 18-item, web-based questionnaire to all 368 PCPs who had the option of referring to SFGH.
Measurements
We asked participants to rate time spent submitting a referral, guidance of workup, wait times, and change in overall clinical care compared to prior referral methods using 5-point Likert scales. We used multivariate logistic regression to identify variables associated with perceived improvement in overall clinical care.
Results
Two hundred ninety-eight PCPs (81.0%) from 24 clinics participated. Over half (55.4%) worked at hospital-based clinics, 27.9% at county-funded community clinics, and 17.1% at non-county-funded community clinics. Most (71.9%) reported that electronic referrals had improved overall clinical care. Providers from non-county-funded clinics (AOR 0.40, 95% CI 0.14-0.79) and those who spent ≥6 min submitting an electronic referral (AOR 0.33, 95%CI 0.18-0.61) were significantly less likely than other participants to report that electronic referrals had improved clinical care.
Conclusions
PCPs felt electronic referrals improved health-care access and quality; those who reported a negative impact on workflow were less likely to agree. While electronic referrals hold promise as a tool to improve clinical care, their impact on workflow should be considered.
KEY WORDS: electronic referral, information technology, subspecialty care, safety net health system
INTRODUCTION
Electronic referrals represent an opportunity to use health information technology (health IT) to improve access to subspecialty care. Health IT can improve the safety and efficiency of health care.1 The potential for improvements apply not only to resource-rich settings, but also to safety-net health systems, which the Institute of Medicine characterizes as those that "…offer care to patients regardless of their ability to pay for services, and [for which] a substantial share of their patients are uninsured, Medicaid, or other vulnerable patients."2 While safety-net health systems’ diversity of reimbursement sources may allow for greater innovation in primary care-subspecialist relationships than traditional fee-for-service models,3 safety nets vary in uptake of health IT.4 In addition to a lack of IT resources,5 health systems may encounter barriers because of concerns about the effects on workload, work roles, or workflow.6
In the safety net, the subspecialist shortage is severe.7,8 The under- and uninsured have fewer choices among subspecialists and longer wait times for appointments;9 longer wait times are associated with delays in diagnosis, greater costs, and worse outcomes.10,11 The adoption of electronic referrals could mitigate the effects of the scarcity of subspecialists. Compared to paper referrals, electronic referrals improve the transfer of administrative and clinical information;12 they may reduce duplicate test-ordering,13 and improve both the referring and subspecialty physician’s ability to make treatment decisions.14 These attributes could lead to rational allocation of subspecialty visits, improving clinical outcomes15 while minimizing wasted resources.16-18
Few studies examine the factors that may improve or impede the adoption of electronic referrals in safety net settings.19-21 The literature supports the examination of health IT’s effects on resources and workflow to inform efforts to implement and sustain newer technology.5,22-24 To better understand primary care providers’ experiences with a new electronic referral system for subspecialty care implemented at a public, university-affiliated, teaching hospital [San Francisco General Hospital and Trauma Center (SFGH)], we administered a cross-sectional, web-based survey of all safety net primary care providers who had the option to refer to SFGH for subspecialty care. We hypothesized that, in the opinion of referring providers, electronic referrals would: (1) improve overall clinical care compared to prior methods, (2) improve referring providers’ access to subspecialists for non-urgent patient issues, and that (3) changes in referring clinic workflow would improve referring providers’ satisfaction.
METHODS
Setting
In July 2005, the SFGH gastroenterology (GI) and liver clinics launched an electronic referral system that allowed a gastroenterologist to triage and allocate limited appointments. Other subspecialty clinics subsequently adopted the electronic referral portal: cardiology and pulmonary (January 2007), endocrinology and rheumatology (May 2007), and neurosurgery and orthopedics (July 2007). With each of these clinics, the hospital alerted referring providers that all non-emergent referrals had to be submitted electronically.
Description of Electronic Referral Program
The key attributes of the SFGH electronic referral system include integration of existing electronic health record demographic and clinical data into electronic referrals, centralized triage of referrals by designated subspecialty, and back-and-forth communication between referring providers and a subspecialist reviewer. Referring providers complete an electronic template, to which existing relevant electronic health record information is automatically appended. Subspecialists review requests within 72 h and choose one of the following options: (1) schedule next available regular appointment, (2) schedule an urgent appointment, or (3) do not schedule. When reviewers do not schedule an appointment, they request additional workup or information, or suggest alternative management in lieu of an appointment. The referring and reviewing providers can communicate in an iterative fashion until the reviewer decides to either schedule an appointment or they both agree that the patient does not need one. When the appointment is granted, this decision is electronically transmitted to the clinic scheduler, who makes the appointment. The hospital electronic health record system then generates a letter to the patient and an e-mail to the referring provider alerting him/her to the appointment. The electronic referral portal keeps a database of all submitted referrals, which serves as a tracking mechanism for both referring and subspecialist providers and clinics.
Study Participants
Referrals originate from primary care providers working in one of three safety net clinic systems: (1) SFGH-based primary care clinics (“hospital-based”), (2) community-based, county-funded health centers that share a common electronic health record with SFGH (Community-Oriented Primary Care clinics or “COPC”), and (3) local non-county-funded community health centers (San Francisco Community Consortium Clinics or “Consortium”) whose access to the electronic health record is through a digital firewall. University of California, San Francisco (UCSF) faculty and trainees staff hospital-based clinics, while county-employed providers staff COPC clinics. Individual clinics employ Consortium clinicians. All subspecialist providers are university-employed and hospital-based. In general, UCSF-employed primary care providers have fewer sessions of clinical time per week and care for a smaller panel than COPC and Consortium providers.
Our study included all primary care providers who have the option of referring adult patients to SFGH. We defined primary care providers as either Family Medicine or Internal Medicine physicians or mid-level providers (nurse practitioners or physician assistants) who see adults and practice in primary care clinics.
SURVEY
Survey Method
We developed an 18-item web-based questionnaire based on prior studies and our interest in the domain of impact on clinical care. We chose a priori to pre-test the questionnaire at four sites in order to represent those that had large and small numbers of providers and which demonstrated higher and lower usage of electronic referral. Based on our pre-test, we clarified wording of items and added the domain of impact on clinical practice.
We mailed a letter introducing the study to all eligible participants prior to initiating the survey, and then sent an e-mail to all participants containing a link to the questionnaire. After sending weekly e-mail reminders for 3 weeks, we telephoned and then mailed a paper version to non-responders. We collected questionnaires from October 2007 through January 2008. We offered a light catered lunch to the two clinics with highest response rates.
The institutional review board at University of California, San Francisco approved the study.
Measures of Participant Characteristics
We asked participants to identify their training level (resident, mid-level provider, or attending physician), practice setting (hospital-based, COPC or Consortium), and volume of care (frequency of seeing patients in clinic each week, frequency of using electronic referrals, length of time using electronic referral in months). Because we anticipated that individual preferences for technology would influence providers’ experiences with electronic referrals, we used Prasad and Agarwal’s validated 4-item scale, which asked participants to rate their willingness to use new information technology on a 5-point Likert scale.25
Participant-Specific Process Measures
We asked providers to note when they submitted electronic referrals: “during,” “between,” “after” patient visits, “never, someone else submits for me,” or “never refer.” We defined time spent referring as a categorical variable with five mutually exclusive levels ranging from “less than 2 min from start to submit” to “greater than 10 min from start to submit.”
Measures of Impact on Clinical Care
We asked participants to compare overall clinical care using electronic referrals to prior methods of referring patients to subspecialists on a 5-point Likert scale (“much worse” to “much better than prior methods”).
Measures of Impact on Clinical Practice
We assessed three practice domains: content, process, and access to subspecialists using electronic referrals compared to prior methods. We used a 5-point Likert scale ranging from “much better” to “much worse.” For content measures, participants rated subspecialty guidance of workup and how well the subspecialist addressed the clinical question. For process measures, we asked participants to rate their ability to track the referral. To gauge access to subspecialists, we asked participants to rate wait time for an available appointment for subspecialty clinics, as well as access to a subspecialist for urgent and non-urgent patient issues.
Statistical Analysis
For the main dependent variable “overall clinical care,” we collapsed 5-level Likert scale responses to two levels, “better” (“much” and “somewhat better”) and “not better” (“no change,” “somewhat” and “much worse”). We chose this dichotomization because of our a priori belief that the success of electronic referrals should be measured by its ability to improve clinical care. We tested for bivariate associations and then used a logistic regression model to determine adjusted odds ratios (AOR). We constructed stepwise multiple regression models, considering as candidates all variables that were associated with the outcome at p < 0.20 in bivariate models, using Spearman’s rho for ordinal independent variables and chi-squared tests for dichotomous variables. After constructing the model with the independent variables of “time spent referring” and “affinity for information technology,” we then added other factors singly and in order. We retained the newly added variable if its effect was statistically significant at p < 0.05.
For other dependent variables (i.e., measures of content, process, and access), we were interested in whether things improved, worsened or were left unchanged. Because the results were statistically similar for five versus three categories, we collapsed 5-level responses into three categories: “better” “same” or “worse.”
In Table 1, we present all candidate variables. In Table 2, we present the AOR for the independent variables for overall clinical care. We transferred all responses from a web-based server (DATSTAT Illume 4.5, Seattle, WA) in Excel format to STATA/S.E. 9.2 (Stata Corporation, College Station, TX).
Table 1.
Participant Characteristics (n = 298)
| Characteristic | Number of participants (%) |
|---|---|
| Level of training | |
| Attending physician | 159 (53.5%) |
| Nurse practitioner | 68 (22.9%) |
| Resident | 70 (23.6%) |
| Type of primary care | |
| Internal medicine | 129 (43.3%) |
| Family medicine | 101(33.9%) |
| Primary care (nurse practitioner) | 68 (22.8%) |
| Setting | |
| Hospital-based clinic | 164 (55.0%) |
| County-funded community clinic | 83 (27.9%) |
| Non-county-funded community clinic | 51 (17.1%) |
| Usually submit eReferral | |
| During or between patient visits | 76 (26.6%) |
| After clinic session | 199 (67.0%) |
| Someone else submits for provider* | 19 (6.4%) |
| Minutes spent submitting eReferral | |
| Less than 2 min | 10 (3.5%) |
| 2-5 min | 124 (42.8%) |
| 6-10 minutes | 102 (35.2%) |
| Greater than 10 min | 40 (13.8%) |
| Technophilia scale: 5-point Likert scale (1 = strongly disagree, 3 = neither agree nor disagree, 5 = strongly agree)16 | Mean response (SD) |
| “In general, I tend to…” | |
| Look for ways to experiment with a new information technology (IT) | 3.67 (SD 1.07) |
| First to try out new IT | 3.00 (1.18) |
| Willing to try out new IT | 3.70 (1.09) |
| Like to experiment with new IT | 3.46 (1.12) |
| Summation score | 3.46 (0.93) |
*Nursing or clerical staff submits eReferral for participant
Table 2.
Adjusted Odds Ratios of Physician Report that Clinical Care is Better as a Result of the Electronic Referral Process, by Physician Characteristics
| Bivariate (%) | Multivariate AOR (95% CI)* | |
| Training | ||
| Attending physician | 67.9 | 1.00 |
| Nurse practitioner | 64.6 | 1.30 (0.63-3.75) |
| Resident | 87.1 | 2.31 (0.96-5.54) |
| Type of primary care | ||
| Internal medicine | 73.8 | 1.00 |
| Family medicine | 74.5 | 1.10 (0.56-2.16) |
| Setting | ||
| Hospital-based clinic | 80.9 | 1.00 |
| County-funded community clinic | 67.1 | 0.72 (0.35-1.49) |
| Non-county-funded comm clinic | 50.0 | 0.40 (0.18-0.91)‡ |
| Average time spent submitting referral | ||
| <6 min/eReferral | 83.6 | 1.00 |
| ≥6 min/eReferral | 62.5 | 0.33 (0.18-0.61)§ |
| Affinity for new technology† | ||
| Low affinity | 69.3 | 1.00 |
| High affinity | 74.2 | 1.16 (0.65-2.08) |
“Better” vs. “same” or “worse” with electronic referrals compared to prior method of referring
*Adjusted for training level, specialty, clinic setting, affinity for technology, average minutes spent per eReferral
†5-point Likert scale dichotomized to “high affinity” if summation score of four items ≥3 indicating somewhat or strongly agree with using newer information technology16
‡p = 0.02
§p < 0.01
RESULTS
We collected 298 of 368 questionnaires for an 81.0% response rate. Nearly all (96.0 %) of the participants that responded used the electronic web-based survey; 4% of providers returned the paper version. Two providers worked in more than one clinic, but each of these providers completed a single questionnaire, and their clinics were from within the safety net clinic system (either COPC or Consortium). Less than 3% of data from the main dependent and independent variables were missing; missing data for the other dependent variables ranged from 4.0 to 7.4%. For each model, we restricted the sample size to those with complete data (Table 3).
Table 3.
Participating Clinics by Safety Net Health System (Number of Participants/Number of Eligible Providers at Each Clinic)
| San Francisco Consortium of Community Clinics (“Consortium”) | Community-oriented Primary Care Clinics (“COPC”) | San Francisco General Hospital-based clinics (“hospital-based”) |
|---|---|---|
| Clinic (participants/eligible) | Clinic (participants/eligible) | Clinic (participants/eligible) |
| Curry Senior Center* (4/5) | Castro-Mission Health Center (6/7) | Family Health Center (65/83) |
| Haight-Ashbury Free Medical Clinic (4/7) | Chinatown Health Center (5/6) | General Medicine Clinic (67/87) |
| Glide Health Services (6/13) | Housing Urban Health Clinic (12/12) | Positive Health Program (37/41) |
| Lyon-Martin Health Services (3/3) | Maxine Hall Health Center (6/7) | |
| Mission Neighborhood Health Center (8/15) | Ocean Park Health Center (6/6) | |
| Native American Health Center(3/3) | Potrero Hill Health Center (3/5) | |
| North East Medical Services (9/15) | Silver Ave Family Health Center (5/7) | |
| San Francisco Free Clinic (4/5) | Southeast Health Center (7/8) | |
| South of Market Health Center (7/7) | Tom Waddell Health Center (19/20) | |
| St. Anthony Free Medical Clinic (4/7) | San Francisco Jail Health Services (7/7) | |
| Women’s Community Clinic (1/1) | ||
| Overall participation (%) 65.4 | Overall participation (%) 89.4 | Overall participation (%) 84.0 |
Measures of Participant Characteristics
The majority of participants were attending physicians (53.5%), with residents making up 23.6% and mid-level providers 22.9%. The majority of participants were physicians practicing internal (43.3%) or family medicine (33.9%). Over half (55.4%) of the survey participants worked at hospital-based clinics, while 27.9% worked at COPC, and 17.1% at Consortium clinics. Most participants saw patients fewer than five half-day clinic sessions per week (70.5%) and had been using electronic referrals for more than 6 months (72.7%). A summary score of four items measuring willingness to use newer information technology showed that 63.5% of participants somewhat or strongly agreed that they were open to using new information technology. This “technophilia” did not differ significantly by training level, type of primary care, or clinic setting. The standardized reliability coefficient (Cronbach’s alpha = 0.88) suggests high reliability of the scale.
Process-Specific Participant Measures
Sixty-seven percent of providers submitted electronic referrals at the end of the clinic, a quarter (26.6%) submitted between visits, and 6.4% of participants reported that someone else submitted for them. While 46.3% of participants reported spending 5 min or less submitting an electronic referral, 36.2% spent 6 to 10 min, and 13.8% spent more than 10 min. (Table 1)
Measures of Impact on Clinical Care
Overall, 71.9% of participants felt electronic referrals had improved clinical care; 7.3% felt electronic referrals had worsened care, and 20.7% felt care had not changed (chi-square p < 0.02). In a bivariate analysis comparing improved versus same or worsened care, significantly more of the resident physicians (87.1%) compared to 67.9% of attending physicians and 64.6% of mid-level providers reported that electronic referrals had improved care (p < 0.01). Similar proportions of internists (73.8%) and of family physicians (74.5%) felt that care had improved. Hospital-based primary care providers had the highest proportion of participants who felt care had improved (80.9%), whereas two-thirds of COPC participants (67.1%) and half (50.0%) of Consortium participants did so, representing a significant trend (Spearman’s rho, p ≤ 0.01). In a multivariate analysis adjusted for training level, subspecialty, clinic setting, technophilia, and time spent submitting, we found that working in a Consortium clinic, as compared to a hospital-based clinic, was significantly associated with lower odds of reporting that care had improved (AOR 0.40, 0.18-0.91). Working in a COPC clinic compared to working in a hospital-based clinic was associated with non-significantly reduced odds of reporting that care had improved (AOR 0.72, 0.35-1.49). More than 6 min spent per referral also predicted lower odds of agreeing that clinical care had improved (AOR 0.33, 0.18-0.61), even after adjusting for clinic setting.
Measures of Impact on Clinical Practice
The majority (71.9%) of participants felt that electronic referrals improved guidance of the pre-subspecialty visit workup, while 22.3% reported no change, and 5.0% reported guidance was worse. (Fig. 1) Over half (57.3%) reported that subspecialists addressed the clinical question better, 30.4% reported no change, and 12.9% reported it was worse.
Figure 1.
PCPs’ ratings of attributes of electronic referrals compared to prior referral methods.
Participants reported that electronic referrals had improved overall wait times for subspecialty appointments (53.9%) and access to the subspecialist for non-urgent patient care issues (59.4%). In contrast, 35.1% reported that electronic referrals had improved access to subspecialists for urgent issues, while nearly half (49.5%) reported no change, and 16.1% reported worse access. For these measures on clinical practice, we found no significant subgroup differences.
DISCUSSION
Seventy-two percent of primary care providers reported that electronic referrals improved overall clinical care compared to prior methods. Several factors, such as perceptions of better communication as manifested by improved tracking and increased access manifested by decreased wait times, may have contributed to this impression. We found that nearly 60% of participants reported that access for non-urgent patient issues had improved, supporting our hypothesis that electronic referrals would improve access to care. Participants reported decreased wait times; this perception was born out by administrative data (data not shown). Features that may have enhanced communication were the improvements in tracking, standardized referral templates, and the centralized review. These may have improved the quality and timeliness of information available to subspecialty reviewers through standardization of communication and equitable access to reviewers. Improving the quality and availability of clinical information has been shown to improve the likelihood that the subspecialist responds to the clinical question.14,26,27 Providers had better guidance of the workup, which may have improved effectiveness of the consultation. Improvement in the ability to track referrals may have improved accountability and prevented duplication of efforts.
We were concerned that electronic referrals would disrupt workflow, which could have impaired the perception of improved care. Workflow challenges present important barriers to adoption and sustainability of health IT.28 We found that a significant proportion of Consortium clinic providers reported taking longer to submit electronic referrals than other providers; the length of time for submission was independently associated with lower satisfaction with overall clinical care. The lack of available computer terminals, multiple-step procedures to access the secured electronic referral portal, and frequent disconnections likely contributed to this increased time spent submitting referrals. We did not find that Consortium providers reported less affinity for newer information technology, increasing the likelihood that the finding of increased dissatisfaction was related to structural barriers.
We found evidence of the impact of the “digital divide” among safety-net health systems on acceptability.29 We are attempting to address these barriers by garnering additional resources to improve computer access and connectivity in Consortium clinics and by encouraging those who are able to submit quickly despite the barriers to share their expertise. Yet, the findings within our study point to a larger phenomenon of “digital disparity” in access to and usability of health IT. Our findings are consistent with prior studies showing that interventions using computerized decision support, electronic referrals, or electronic health records have had variable uptake in part due to the heterogeneity of patients, practices, and resources especially among safety net health systems.30
Our study had important limitations. First, we used web-based questionnaires to ask participants about electronic referrals, which may have selected participants who had greater willingness to use IT. However, we mailed paper versions of the questionnaire to non-responders. Second, we relied on primary care provider recall of the referral system prior to electronic referrals, which may introduce recall bias. Third, this study focuses solely on the referring providers’ perspective, and relies on their subjective reports of time spent submitting and wait times. We did not weight responses by clinic setting because clinic setting was not the only way to categorize the variability in our participants. We may, however, have underestimated the discontent among non-respondents, as those from the Consortium were both less likely to report that electronic referrals had improved clinical care and were less likely to respond. Because we were interested in the acceptability to individual participants, we did not adjust responses for panel size or number of referrals per participant. Fourth, we analyzed results from providers referring to a single hospital in a safety net health system, including clinician investigators and residents who, while working full time, spend only a few half-days a week of engaged in direct outpatient care. However, because we included primary care providers from three different health systems, our findings may generalize to other community- and university-based providers who work in safety net settings.
Electronic referrals offer the opportunity to improve the effectiveness and efficiency of subspecialty care without inappropriately restricting access. Integrated health-care delivery systems that do not rely on fee-for-service reimbursement may be fertile ground for attempts to improve the efficiency of subspecialty care, thus improving access.31,32 We found that digital disparities among safety net health systems depend less on individual provider preferences than on access to and the usability of health IT. Future interventions should incorporate primary care provider and referring clinic experiences in evaluating the adoption and spread of electronic referrals. In our study, safety net primary care providers reported that electronic referrals improved not only access to specialty care, but also the overall clinical care of patients. Our findings suggest that IT innovations can be instrumental for safety net settings in attaining the “holy grail”33 of ensuring access to timely, high-quality subspecialty care.
Acknowledgements
We thank the providers, staff, and administration at the San Francisco Community Clinic Consortium, the Community Oriented Primary Care clinics, and San Francisco General Hospital for their time and assistance in completing the questionnaires. We acknowledge Anita Stewart, PhD, for her guidance in designing the survey instrument and Peter Bacchetti, PhD, for his statistical advice. We also thank Susan Straus, PhD, for her recommendation to include a previously validated instrument to measure affinity for information technology. The abstract for this paper was previously presented as a poster at the Society of General Internal Medicine Annual Meeting in April 2008. Drs. Kushel, Chen, and Yee and Ms. Keith received support from the San Francisco Health Plan (eReferral Spread Project 11/1/08–6/30/08). Dr. Yee was partially supported by the William and Mary Ann Rice Memorial Distinguished Professorship. Dr. Kim was in the Division of General Internal Medicine at UCSF/San Francisco General Hospital when this work was completed; she was supported by a Health Resources Services Administration fellowship training grant (no. T32 HP19025). The funding agencies did not play a role in designing the study, interpreting the results, or writing the manuscript.
Conflict of Interest None disclosed.
References
- 1.Wagner EH. Chronic disease management: what will it take to improve care for chronic illness? Eff. Clin. Pract. 1998;1:2-4. [PubMed]
- 2.Ein Lewin M, Altman SH. America's health care safety net: intact but endangered. Washington, DC: Institute of Medicine; 2000. [PubMed]
- 3.Smith SM, Allwright S, O'Dowd T. Effectiveness of shared care across the interface between primary and specialty care in chronic disease management. Cochrane Database Syst Rev 2007:CD004910. [DOI] [PubMed]
- 4.Fischer MA, Vogeli C, Stedman MR, Ferris TG, Weissman JS. Uptake of electronic prescribing in community-based practices. J Gen Intern Med 2008;23:358-63. [DOI] [PMC free article] [PubMed]
- 5.Leu MG, Cheung M, Webster TR, et al. Centers speak up: the clinical context for health information technology in the ambulatory care setting. J Gen Intern Med 2008;23:372-8. [DOI] [PMC free article] [PubMed]
- 6.Bodenheimer T, Grumbach K. Electronic technology: a spark to revitalize primary care? Jama 2003;290:259-64. [DOI] [PubMed]
- 7.Felt-Lisk S, McHugh M, Howell E. Monitoring local safety-net providers: do they have adequate capacity? Health Aff (Millwood) 2002;21:277-83. [DOI] [PubMed]
- 8.Gusmano MK, Fairbrother G, Park H. Exploring the limits of the safety net: community health centers and care for the uninsured. Health Aff (Millwood) 2002;21:188-94. [DOI] [PubMed]
- 9.Felland LE, Felt-Lisk S, McHugh M. Health care access for low-income people: significant safety net gaps remain. Issue Brief Cent Stud Health Syst Change 2004:1-4. [PubMed]
- 10.Knudtson ML, Beanlands R, Brophy JM, Higginson L, Munt B, Rottger J. Treating the right patient at the right time: access to specialist consultation and non-invasive testing. Can J Cardiol 2006;22:819-24. [DOI] [PMC free article] [PubMed]
- 11.Levin A. Consequences of late referral on patient outcomes. Nephrol Dial Transplant 2000;15(Suppl 3):8-13. [DOI] [PubMed]
- 12.Shaw LJ, de Berker DA. Strengths and weaknesses of electronic referral: comparison of data content and clinical value of electronic and paper referrals in dermatology. Br J Gen Pract 2007;57:223-4. [PMC free article] [PubMed]
- 13.Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. Jama 1998;280:1339-46. [DOI] [PubMed]
- 14.Medow MA, Wilt TJ, Dysken S, Hillson SD, Woods S, Borowsky SJ. Effect of written and computerized decision support aids for the U.S. agency for health care policy and research depression guidelines on the evaluation of hypothetical clinical scenarios. Med Decis Making 2001;21:344-56. [DOI] [PubMed]
- 15.Prentice JC, Pizer SD. Delayed access to health care and mortality. Health Serv Res 2007;42:644-62. [DOI] [PMC free article] [PubMed]
- 16.Reinhardt UE. The United States: breakthroughs and waste. J Health Polit Policy Law 1992;17:637-66. [DOI] [PubMed]
- 17.Gruen RL, Weeramanthri TS, Knight SE, Bailie RS. Specialist outreach clinics in primary care and rural hospital settings. Cochrane Database Syst Rev 2004:CD003798. [DOI] [PMC free article] [PubMed]
- 18.Madhok R, Green S. Orthopaedic outpatient referral guidelines: experience in an English health district. Int J Qual Health Care 1994;6:73-6. [DOI] [PubMed]
- 19.Eccles M, Steen N, Grimshaw J, et al. Effect of audit and feedback, and reminder messages on primary-care radiology referrals: a randomised trial. Lancet 2001;357:1406-9. [DOI] [PubMed]
- 20.Abbott KC, Mann S, DeWitt D, Sales LY, Kennedy S, Poropatich RK. Physician-to-physician consultation via electronic mail: the Walter Reed Army Medical Center Ask a Doc system. Mil Med 2002;167:200-4. [PubMed]
- 21.Deutscher D, Hart DL, Dickstein R, Horn SD, Gutvirtz M. Implementing an integrated electronic outcomes and electronic health record process to create a foundation for clinical practice improvement. Phys Ther 2008;88:270-85. [DOI] [PubMed]
- 22.Bodenheimer T. Coordinating care–a perilous journey through the health care system. N Engl J Med 2008;358:1064-71. [DOI] [PubMed]
- 23.Zandieh SO, Yoon-Flannery K, Kuperman GJ, Langsam DJ, Hyman D, Kaushal R. Challenges to EHR implementation in electronic- versus paper-based office practices. J Gen Intern Med 2008;23:755-61. [DOI] [PMC free article] [PubMed]
- 24.Crosson JC, Isaacson N, Lancaster D, et al. Variation in electronic prescribing implementation among twelve ambulatory practices. J Gen Intern Med 2008;23:364-71. [DOI] [PMC free article] [PubMed]
- 25.Agarwal R, Prasad J. A conceptual and operational definition of personal innovativeness in the domain of information technology. Information Systems Research 1998;9:204-15. [DOI]
- 26.Wright J, Harrison S, McGeorge M, et al. Improving the management and referral of patients with transient ischaemic attacks: a change strategy for a health community. Qual Saf Health Care 2006;15:9-12. [DOI] [PMC free article] [PubMed]
- 27.Gandhi TK, Sittig DF, Franklin M, Sussman AJ, Fairchild DG, Bates DW. Communication breakdown in the outpatient referral process. J Gen Intern Med 2000;15:626-31. [DOI] [PMC free article] [PubMed]
- 28.Moiduddin A, Gaylin D. Health information technology adoption among health centers. A digital divide in the making? National Health Policy Forum 2007:17.
- 29.Tierney WM, Rotich JK, Smith FE, Bii J, Einterz RM, Hannan TJ. Crossing the "digital divide:" implementing an electronic medical record system in a rural Kenyan health center to support clinical care and research. Proc AMIA Symp 2002:792-5. [PMC free article] [PubMed]
- 30.Faulkner A, Mills N, Bainton D, et al. A systematic review of the effect of primary care-based service innovations on quality and patterns of referral to specialist secondary care. Br J Gen Pract 2003;53:878-84. [PMC free article] [PubMed]
- 31.Grumbach K, Osmond D, Vranizan K, Jaffe D, Bindman AB. Primary care physicians' experience of financial incentives in managed-care systems. N Engl J Med 1998;339:1516-21. [DOI] [PubMed]
- 32.Ludwick DA, Doucette J. Primary Care Physicians' Experience with Electronic Medical Records: Barriers to Implementation in a Fee-for-Service Environment. Int J Telemed Appl 2009;2009:853524. [DOI] [PMC free article] [PubMed]
- 33.Skinner AC, Mayer ML. Effects of insurance status on children's access to specialty care: a systematic review of the literature. BMC Health Serv Res 2007;7:194. [DOI] [PMC free article] [PubMed]