Abstract
Background
Limited access to specialist care remains a major barrier to health care in Canada, affecting patients and primary care providers alike, in terms of both long wait times and inequitable availability. We developed an electronic consultation system, based on a secure web-based tool, as an alternative to face-to-face consultations, and ran a pilot study to evaluate its effectiveness and acceptability to practitioners.
Methods
In a pilot program conducted over 15 months starting in January 2010, the e-consultation system was tested with primary care providers and specialists in a large health region in Eastern Ontario, Canada. We collected utilization data from the electronic system itself (including quantitative data from satisfaction surveys) and qualitative information from focus groups and interviews with providers.
Results
Of 18 primary care providers in the pilot program, 13 participated in focus groups and 9 were interviewed; in addition, 10 of the 11 specialists in the program were interviewed. Results of our evaluation showed good uptake, high levels of satisfaction, improvement in the integration of referrals and consultations, and avoidance of unnecessary specialist visits. A total of 77 e-consultation requests were processed from 1 Jan. 2010 to 1 Apr. 2011. Less than 10% of the referrals required face-to-face follow-up. The most frequently noted benefits for patients (as perceived by providers) included improved access to specialist care and reduced wait times. Primary care providers valued the ability to assist with patient assessment and management by having access to a rapid response to clinical questions, clarifying the need for diagnostic tests or treatments, and confirming the need for a formal consultation. Specialists enjoyed the improved interaction with primary care providers, as well as having some control in the decision on which patients should be referred.
Interpretation
This low-cost referral system has potential for broader implementation, once payment models for physicians are adapted to cover e-consultation.
Primary care providers manage a broad range of patients’ health issues, with many providers requiring specialist support at some point. In Ontario alone, 54 000 patients are referred to specialists daily, the majority of these referrals being requested by primary care providers.1
Traditionally, when a consultation is needed, the primary care provider sends the patient for a formal face-to-face consultation with the specialist. However, such appointments may be delayed because of lack of specialists’ availability, and the patient is frequently required to travel some distance.
Waiting for specialist care remains the most important barrier to health care access in Canada.2 The average wait time to see a specialist after being referred by a primary care provider has increased substantially over the past 2 decades, from 3.7 weeks in 1993 to 8.9 weeks in 2010.3 After being seen by a specialist, Canadian patients wait, on average, another 9.3 weeks to receive treatment. 3 The complexity of the health care system and the many steps involved in a specialist consultation also contribute to the overall length of the process (see Figure 1).4
Figure 1.
Steps in process of care involving both a primary care provider (PCP) and a specialist. Based on a prototype shared with the authors by the College of Family Physicians of Canada and a model proposed by the Institute for Clinical Evaluative Sciences.
Some primary care physicians use informal (so-called curbside) consultations to ask questions of specialists they know and meet by chance. However, opportunities for such informal consultations with specialists have been reduced, now that many primary care providers work completely outside the hospital setting.5,6 In addition, concerns exist about the quality and adequacy of information exchanged during informal consultations.6-9 Alternatively, a consultation may be done informally by telephone or e-mail.7,10,11
The use of e-mail systems for consultation has been explored in a variety of settings, including military medical centres and clinical teaching units.12-15 However, confidentiality, privacy, and the security of e-mail systems are areas of concern11 that have limited the potential expansion of such consultation systems.
Telemedicine is another alternative to traditional consultation, especially in remote areas.16 However, telemedicine is not widely used, and the requirement for specialized and often expensive equipment makes it inaccessible to most primary care providers. Furthermore, studies have shown that the overall cost-effectiveness of telemedicine is limited.17 In addition, it requires the simultaneous presence of the patient, the primary care provider, and the specialist.6,18
We were interested in the feasibility of establishing an electronic consultation (e-consultation) system and in exploring the impacts and potential benefits of such a system for patients and their health care providers. Using an existing multipurpose virtual collaboration space, we developed and piloted an electronic consultation system with primary care providers and specialists in Eastern Ontario. Use of a secure, region-wide network allowed for ease of access (similar to e-mail) while fulfilling all privacy requirements. Here, we report the findings of our evaluation of the pilot phase, which ran from 1 Jan. 2010 to 1 Apr. 2011.
Methods
Setting and participants
The study took place in the Champlain Local Health Integration Network (LHIN), 1 of 14 regional health districts in Ontario, encompassing Ottawa and its surrounding communities. The Champlain LHIN serves a culturally diverse region with a population of 1.2 million people whose chronic disease burdens and health outcomes are comparable to those of Ontario and the rest of Canada.19 The area has one main urban centre with a large, multisite tertiary care hospital housing many of the specialty services, which provide care to people living in outlying rural communities.
Participants in the pilot study consisted of 29 health care providers (14 family physicians, 4 nurse practitioners, and 11 specialists), all of whom were invited to give feedback in an interview or focus group (or both).
We used a mixed-methods exploratory embedded design in which one data type supports or provides a secondary role in a study based primarily on another data type. The rationale for this approach was that we were seeking answers to multiple questions for which a single data type would have been insufficient.20 More specifically, the qualitative methods used in this study, which consisted of interviews and focus groups, served as the primary means of obtaining attitudinal information about practitioners’ experience with the e-consultation service. The quantitative methods, based on system utilization data, augmented the qualitative findings and provided information about practitioners’ actual use of the e-consultation service.
E-consultation service
The e-consultation service was built upon the secure LHIN Collaboration space hosted at the Winchester District Memorial Hospital in Winchester, Ontario. This web-based tool was developed and deployed for e-scheduling applications and other interactions among health care workers in the region. The e-consultation applications, including associated forms, automated work flows, and interface engines, were based on off-the-shelf components that many health care organizations across Canada were already using, an approach that we anticipated would simplify replication and deployment of the system in other regions. The research team developed the forms (sample shown in Appendix A) and work flow (diagram shown in Figure 2) for this project in consultation with the participants, without consulting any vendors. We conducted both a privacy impact assessment and a threat risk assessment in compliance with Ontario’s Personal Health Information Protection Act. We also consulted the Canadian Medical Protective Association about medical risks and liability issues related to use of the e-consultation system. The duty of care for consultations requested and provided in the course of this project was no different than for any other consultation, such as a telephone consultation or a “corridor consultation.” In fact, the e-consultation system was potentially safer than various informal modes of consultation, because the requests were automatically recorded and the specialist had the option of requesting more information or declining to provide a recommendation if there was insufficient information.
Figure 2.
Work flow for e-consultation service involving primary care providers (PCPs) and specialists
The e-consultation system allowed primary care providers to submit a patient-specific clinical question to a specialist, using the standardized web-based form. Each provider had a unique user name and password for logging on and could access the web forms from any computer with high-speed Internet access. Supplementary patient information, such as laboratory test results, digital images, and health history, could be included with the request, to assist the specialist in making an informed recommendation. Each consultation request was assigned by project staff to an appropriate specialist (according to the specialty required and specialists’ availability), who was given 1 week to respond. Depending on the request, the specialist had the following options:
provide answers to questions and avoid the need for a patient visit
request additional information before providing advice
recommend a formal referral, in which case additional diagnostic tests or courses of treatment could be requested and started before the appointment
Both the specialist and the primary care provider received e-mail notification at each stage of the process, so they knew the status of the e-consultation and could log back into the system to see the response and/or provide more information as needed. In addition, a permanent record of the e-consultation was created, which could be downloaded into the patient’s health record.
Training
The system was easy to use and began with self-registration at the site. Each registration was approved centrally once the provider’s identity was confirmed. Training consisted of a 20-minute session, which could be done in person or by telephone, depending on the provider’s level of comfort with the technology. Participants were given a training manual for future reference and a “cheat sheet,” a card summarizing the steps to completing a referral (Appendix B). Ongoing technical and system support was provided by the Champlain LHIN.
Qualitative data: focus groups and interviews
For focus groups and interviews involving primary care providers, we used purposeful sampling with maximum variation, seeking to include providers from all sites, as described by Patton.21 Because of the smaller number of specialists who participated in the pilot study, we conducted qualitative interviews (but no focus groups) with all of the specialist participants, except for one, who had scheduling conflicts.
We started with 2 focus groups for all participating primary care providers, each lasting about 90 minutes. Each participant reviewed and signed a consent form at the beginning of the focus group and received $300 for participation. A moderator’s guide was used consistently across focus groups. Focus group discussions were audiotaped and transcribed verbatim. A research consultant (M.S.R.) then conducted more in-depth semi-structured telephone interviews, each about 60 minutes in length, with primary care providers who met the criterion of being active users (i.e., participating in at least one e-consultation ). In addition, all but one of the specialists were interviewed by the same research consultant. These interviews were similar, in terms of length and content, to the interviews with primary care providers. All of the interviews were audiotaped and transcribed verbatim.
The research consultant, an experienced qualitative researcher, developed the initial coding framework (which was based on the interview questions and the moderator’s guide), coded the transcripts from the focus groups and interviews, and worked with other members of the research team to further refine the coding categories, which were then applied to all transcripts. Data were analyzed using NVIVO8 software (QSR International [Americas] Inc., Burlington, Mass.), which identified themes by reviewing the data within and across codes. Coding summaries were reviewed by 2 other members of the research team. Inconsistencies were resolved through discussion until consensus was reached within the team. Qualitative information was compared across participant groups and against the quantitative information (described below) to help in interpretation of the overall findings.
Ethics approval for this study was obtained from the Ottawa Hospital Research Ethics Board. All primary care physicians and specialists provided informed consent before participating in the study. Primary care providers were asked to obtain each patient’s verbal consent before using the e-consultation system for that patient’s care.
Quantitative data: system utilization data
For each e-consultation, data regarding the primary care provider, the consulting specialist, the clinical questions posed, and the answers provided were automatically stored by the system. Log-in time, time spent on the consultation, time for reply and closure of the case, and responses to a satisfaction survey were all collected and stored permanently on the system.
Results
Participant characteristics
Of the 18 potential participants in focus groups for primary care providers, 13 agreed to participate (7 in focus group 1 and 6 in focus group 2). The focus group participants included both family physicians (n = 9) and nurse practitioners (n = 4) from 2 rural primary care practices, both of which were using electronic medical record systems. Nine primary care providers met the criterion of being active users (i.e., participated in at least one e-consultation) and participated in an interview. Three of these interview participants had also participated in a focus group. All 9 were using a computer or web-based device to assist in their daily practice (before initiation of the e-consultation service). Eight of these providers practised in a rural location, and one in a semi-rural location; 2 providers worked in a solo practice and 7 in a group practice. The median number of years in practice was 20 (range 2–27.5 years). The average number of usual referrals per week was 15. In addition, 10 specialists covering 7 specialties participated in an interview, all of whom were using a computer or web-based device to assist in their daily practice. Nine of the specialists were in urban locations (academic practice), and 1 was in a rural location (community practice). The median number of years in practice for specialists was 25 (range 4–34 years).
Use of e-consultation
From 1 Jan. 2010 to 1 Apr. 2011, 77 e-consultation requests were made through the system by 18 primary care providers, with a response from 1 of the 11 participating specialists. Most e-consultation requests were addressed to a dermatologist (19 [25%]), an endocrinologist (18 [23%]), a cardiologist (9 [12%]), or a rheumatologist (9 [12%]). Most responses (58 or 75%) were received within the specified time frame of 1 week, with 19 (25% of the total) being completed within 1 day. The average response time was 5.5 days. Less than 10% of the e-consultations required follow-up face-to-face visits. In those cases, the specialist often asked the primary care provider to take further action (such as advising the patient, ordering diagnostic tests, or starting a course of treatment) while the patient was waiting for his or her consultation.
Satisfaction
The majority of physicians and nurse practitioners were satisfied with the e-consultation service. Many physicians commented on its simplicity and effectiveness. One primary care provider (ID 205) stated, “It is pretty straightforward. I typed in my consult and sent if off.” One specialist (ID 101) said, “I was absolutely satisfied with the type of information, the clarity of it, and especially the timeliness of it.”
Some specialists mentioned how useful it was to build their work on what another physician had already done. Noted one (ID 202), “It’s always quicker to read someone’s findings rather than to go ahead and do the full exam yourself. I probably would spend anywhere from 30 to 45 minutes with a new patient. What I reported as having spent on e-consultation was much less than that. Nothing more than 20 minutes.” All specialists and most primary care providers suggested that the e-consultation service be expanded in the future, and many indicated that they would recommend e-consultation to a colleague.
Perceived or potential benefits
The perceived or potential benefits of e-consultation, as reported by pilot participants, applied to 4 main groups: patients, primary care physicians, specialists, and the healthcare system as a whole (Table 1).
Table 1.
Perceived benefits of e-consultation, as reported by participating physicians
For patients, the most frequently noted benefits, as perceived by study participants, included improved access to specialist care by avoidance of the need for face-to-face consultation or re-consultation, as well as reduced delays or provision of timely access. Both groups of providers commented on the value of the 2-way communication facilitated by the e-consultation system, which allowed efficient triage of cases and rapid clarification of clinical questions and responses. Primary care providers valued the ability to assist with patient assessment and management by responding to clinical questions, clarifying the need for diagnostic tests or treatments, and confirming the need for a formal consultation. Specialists enjoyed receiving comprehensive advance work on a case before the consultation, as this allowed them to better understand the patient’s history and any previous investigative work done, as well as giving them some decision-making control about who should be referred.
Areas of concern
Users of the e-consultation system commented on a few areas of concern during this pilot phase.
The impersonal nature of communication through an electronic system was one concern. One specialist (ID 205) noted, “We are always worried about giving advice over the phone. And, in this case, we are giving advice having not seen or examined the patient, everything being based on the information that has been posted [and] our working knowledge.” Although primary care providers often seek particular specialists for regular referrals because of familiarity, comfort or an existing working relationship, this mode of selecting a specialist was not available to participants in the e-consultation pilot system. Consequently, lack of comfort in using the system stemmed partly from primary care providers not knowing the specialist with whom they were consulting. Specialists also welcomed the prospect of making e-consultations more personal by having the system reveal the identity of the requesting primary care provider.
Participants also commented on technical issues related to use of the system, such as variability in typing skill and in ability to learn and adopt the new technology. Other concerns in this area were related to privacy and security, use of online forms, and technical quality of attachments.
Suggestions for the future
The participants provided suggestions for a larger-scale e-consultation service, including adaptation of work flow processes for referrals to account for higher volumes of e-consultation. Most primary care providers suggested that the integration of e-consultation into electronic medical records would be a positive next step. All respondents envisioned a role for an “e-consultation assistant” to help with such tasks as triaging cases, booking appointments when needed, scheduling e-consultations, and taking photographs.
Furthermore, some form of hands-on training for initial set-up and access to a resource person were desired by novice users, aspects that could potentially be subsumed under the e-consultation assistant role. As one participant said, “I am wondering if I can get [trained on] how to implement the e-consult in my office or if we can go ‘get some training [on] how to implement the e-consult through [electronic medical record software] by direct link’” (FG2).
Interpretation
We developed and implemented an effective e-consultation system for use within our health region. Our approach was feasible and highly acceptable to both the specialists and the primary care providers who participated in our pilot study. The use of existing infrastructure and the development of a system based on off-the-shelf components of a commonly used software product means that this system could be broadly implemented at low cost. Although many requests for specialist advice and consultations for procedures will continue to require a face-to-face consultation, even a 10% reduction in the number of face-to-face consultations would be significant, translating to 5400 fewer consultations a day in Ontario alone.1 Such a reduction could reduce overall wait times for patients who do require face-to-face consultations and could reduce patients’ costs (e.g., for time off work and travel). The estimated cost savings, even with specialists’ fees for e-consultations factored in, could be as much as $400 000/day (assuming that the cost of a usual visit is $150 and the cost of an e-consultation is half that [i.e., $75]).
The e-consultation system that we developed and implemented addressed many of the limitations and barriers reported with other e-consultation systems, such as high cost; needs for special equipment, synchronous scheduling of appointments, and extensive training; and privacy and security issues.[11,12,15,22,23] We overcame these drawbacks by developing a system that was based on existing secure private infrastructure, that required less than 30 minutes of training, that was very convenient for providers, and that involved asynchronous communication.
The evaluation results were encouraging, with most participants (both primary care providers and specialists) being very satisfied with their experience of using the e-consultation system. Engagement and adoption of new technology by health care providers can be challenging. It has been shown that physicians who see innovation as offering a relative advantage over current practice will more readily commit to and adopt innovations than those who see no advantage.24 As perceived by our participants, the use of e-consultation resulted in several benefits for patients, providers, and the health care system, including a considerable decrease in the need to refer primary care patients to a specialist for an in-person visit. Both primary care providers and specialists saw much value in expanding the e-consultation system.
Several issues would have to be addressed in moving beyond the pilot stage of this project, including greater organization of specialty care and better integration of the e-consultation system into regular work flows for all providers. A population-based approach to delivery of specialty services would include the creation of “groups” of specialists who see their mandate as providing whole-population specialty care, as opposed to the current system of individual, referral-based demand.
Looking forward, there is the question of how to incorporate into this e-consultation system some of the best features of existing systems, such as the face-to-face capacity of telemedicine and the ability to deal with urgent questions by telephone. Furthermore, there is the larger issue of achieving system-wide integration, whereby e-consultation is linked with other health information systems, such as regional diagnostic centres and laboratories, and even the possibility of using e-consultation as a stepping stone to an e-referral system.
Limitations
This study was limited by its sampling bias. More specifically, the sampling for interviews involved providers who were most interested in participating in the e-consultation pilot. Most of the primary care providers were from a single group practice in a rural setting, and most specialists had an academic practice in an urban setting. The results may not be generalizable to other specialist services not included in the study. Future studies should explore the experiences of a wider sample of participants. The addition of patient interviews would also strengthen the evaluation.
Current status of e-consultation service
We have continued to develop the e-consultation system, including (in response to feedback from the pilot study participants) the creation of a self-registration system, simplification of the instructions for use, and automatic identification of the provider within the e-consultation request. At the time of publication, in late 2012, we were addressing sustainability issues related to funding mechanisms for e-consultation. As of 1 Nov. 2012, a total of 140 family doctors, 31 nurse practitioners, and 40 specialists across 18 specialties were registered users. By the same date, a total of 644 consultations had been completed.
Conclusion
This study revealed actual and potential benefits of e-consultation for patients, providers, and the health care system. By implementing this service, we improved access to specialty care, while ensuring that high-quality advice was transmitted securely, which led to overall cost savings for the health care system.
Acknowledgments
We thank all of the physicians who piloted the system and provided invaluable feedback for the project. We also thank the technical support and development team from Winchester Memorial Hospital (Sean O’Brien) and the Champlain Local Health Integration Network (Glenn Alexander).
Biographies
Clare Liddy, MD, MSc, is an Assistant Professor in the Department of Family Medicine, University of Ottawa, and a Clinician Investigator with the C.T. Lamont Primary Health Care Research Centre, Bruyère Research Institute, Ottawa, Ontario.
Margo S. Rowan, PhD, is an Assistant Professor in the Department of Family Medicine, University of Ottawa, and is the Principal Lead at Rowan Research and Evaluation, Ottawa, Ontario.
Amir Afkham, BEng, is a Senior Project Manager with the Champlain Local Health Integration Network, Ottawa, Ontario.
Julie Maranger, BScN, is a Research Nurse in the Division of Endocrinology and Metabolism, The Ottawa Hospital, and is a Clinical Research Coordinator with the Ottawa Hospital Research Institute, Ottawa, Ontario.
Erin Keely, MD, is Chief, Division of Endocrinology and Metabolism at The Ottawa Hospital; Professor in the Departments of Medicine and of Obstetrics and Gynecology, University of Ottawa; and a Clinician Scientist at the Ottawa Hospital Research Institute, Ottawa, Ontario.
Appendices
Appendix A.
Sample forms used in the web-based e-consultation system for primary care providers and specialists.
Appendix B.
Card summarizing the steps to completing a referral, for use by participants in the pilot program.
Footnotes
Competing interests: None declared.
Funding source: This study was supported by The Ottawa Hospital Academic Medical Organization Innovation Fund, eHealth Ontario and the Champlain Local Health Integration Network.
Contributors: Clare Liddy and Erin Keely originally conceived the idea for this study, developed the methods, wrote the first draft of the manuscript, and implemented the project. Margo Rowan led the collection and analysis of qualitative data. All authors participated in study design, data analysis, and critical review and final approval of the manuscript.
References
- 1.Manuel D G, Maaten S, Thiruchelvam D, Jaakkimainen L, Upshur R. Primary care in the health care system. In: Jaakkimainen L, Upshur R, Klein-Geltink J, Leong A, Maaten S, Schultz S, Wang L, editors. Primary care in Ontario: ICES atlas. Toronto (ON): Institute for Clinical Evaluative Sciences; 2006. pp. 1–14. http://www.ices.on.ca/file/PC_atlas_prelims_complete.pdf. [Google Scholar]
- 2.Berthelot J M, Sanmartin C. Access to health care services in Canada, January to December 2005. Ottawa (ON): Statistics Canada; 2006. Report No.: 82-575-XIE. [Google Scholar]
- 3.Barua B, Rovere M, Skinner B J. Waiting your turn: wait times for health care in Canada. 2010 report. 20th ed. Calgary (AB): Fraser Institute; 2010. [Google Scholar]
- 4.Tu J V, Pinfold S P, McColgan P, Laupacis A. Access to health services in Ontario. Toronto (ON): Institute for Clinical Evaluative Sciences; 2006. [Google Scholar]
- 5.Chan Benjamin T B. The declining comprehensiveness of primary care. CMAJ. 2002 Feb 19;166(4):429–434. http://pubmedcentralcanada.ca/pmcc/articles/pmid/11876170. [PMC free article] [PubMed] [Google Scholar]
- 6.Horner Kathryn, Wagner Ed, Tufano Jim. Electronic consultations between primary and specialty care clinicians: early insights. Issue Brief (Commonw Fund) 2011;23:1–14. [PubMed] [Google Scholar]
- 7.Bergus G R, Sinift S D, Randall C S, Rosenthal D M. Use of an e-mail curbside consultation service by family physicians. J Fam Pract. 1998;47(5):357–360. http://www.scholaruniverse.com/ncbi-linkout?id=9834770. [PubMed] [Google Scholar]
- 8.Manian F A, Janssen D A. Curbside consultations. A closer look at a common practice. JAMA. 1996;275(2):145–147. doi: 10.1001/jama.1996.03530260059032. http://jama.jamanetwork.com/article.aspx?doi=10.1001/jama.1996.03530260059032. [DOI] [PubMed] [Google Scholar]
- 9.Kuo D, Gifford D R, Stein M D. Curbside consultation practices and attitudes among primary care physicians and medical subspecialists. JAMA. 1998;280(10):905–909. doi: 10.1001/jama.280.10.905. http://jama.ama-assn.org/cgi/doi/10.1001/jama.280.10.905. [DOI] [PubMed] [Google Scholar]
- 10.Borowitz S M, Wyatt J C. The origin, content, and workload of e-mail consultations. JAMA. 1998 Oct 21;280(15):1321–1324. doi: 10.1001/jama.280.15.1321. http://www.scholaruniverse.com/ncbi-linkout?id=9794310. [DOI] [PubMed] [Google Scholar]
- 11.Using email communication with your patients: legal risks. Ottawa (ON): Canadian Medical Protective Association; 2005. [revised 2009]. Report No. IS0586-E https://www.cmpa-acpm.ca/cmpapd04/docs/resource_files/infosheets/2005/com_is0586-e.cfm. [Google Scholar]
- 12.Abbott Kevin C, Mann Scott, DeWitt Daisy, Sales Linda Y, Kennedy Sean, Poropatich Ron K. Physician-to-physician consultation via electronic mail: the Walter Reed Army Medical Center Ask a Doc system. Mil Med. 2002;167(3):200–204. [PubMed] [Google Scholar]
- 13.Angstman Kurt B, Adamson Steven C, Furst Joseph W, Houston Margaret S, Rohrer James E. Provider satisfaction with virtual specialist consultations in a family medicine department. Health Care Manag (Frederick) 2009;28(1):14–18. doi: 10.1097/HCM.0b013e318196def8. [DOI] [PubMed] [Google Scholar]
- 14.Neufeld Jonathan D, Yellowlees Peter M, Hilty Donald M, Cobb Hattie, Bourgeois James A. The e-Mental Health Consultation Service: providing enhanced primary-care mental health services through telemedicine. Psychosomatics. 2007;48(2):135–141. doi: 10.1176/appi.psy.48.2.135. http://www.scholaruniverse.com/ncbi-linkout?id=17329607. [DOI] [PubMed] [Google Scholar]
- 15.Stoves John, Connolly John, Cheung Chee K, Grange Angela, Rhodes Penny, O'Donoghue Donal, Wright John. Electronic consultation as an alternative to hospital referral for patients with chronic kidney disease: a novel application for networked electronic health records to improve the accessibility and efficiency of healthcare. Qual Saf Health Care. 2010;19(5):54. doi: 10.1136/qshc.2009.038984. [DOI] [PubMed] [Google Scholar]
- 16.Hjelm N M. Benefits and drawbacks of telemedicine. J Telemed Telecare. 2005;11(2):60–70. doi: 10.1258/1357633053499886. http://jtt.rsmjournals.com/cgi/doi/10.1258/1357633053499886. [DOI] [PubMed] [Google Scholar]
- 17.Jacklin P B, Roberts J A, Wallace P, Haines A, Harrison R, Barber J A, Thompson S G, Lewis L, Currell R, Parker S, Wainwright P. Virtual outreach: economic evaluation of joint teleconsultations for patients referred by their general practitioner for a specialist opinion. BMJ. 2003;327(7406):84. doi: 10.1136/bmj.327.7406.84. http://pubmedcentralcanada.ca/pmcc/articles/pmid/12855528. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Wallace S, Wyatt J, Taylor P. Telemedicine in the NHS for the millennium and beyond. Postgrad Med J. 1998 Dec 1;74(878):721–728. doi: 10.1136/pgmj.74.878.721. http://pubmedcentralcanada.ca/pmcc/articles/pmid/10320886. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Bains N, Dall K, Hay C, Pacey M, Sarkella J, Ward M. Population health profile: Champlain LHIN. Toronto (ON): Ministry of Health and Long-Term Care; 2008. Document no. HSIP-PHP-01. [Google Scholar]
- 20.Creswell J W, Plano Clark V L. Designing and conducting mixed methods research. Thousand Oaks (CA): Sage; 2007. [Google Scholar]
- 21.Patton M Q. Qualitative research and evaluation methods. 3rd ed. Thousand Oaks (CA): Sage; 2002. [Google Scholar]
- 22.Jaatinen Pekka T, Aarnio Pertti, Remes Jouko, Hannukainen Jorma, Köymäri-Seilonen Tarja. Teleconsultation as a replacement for referral to an outpatient clinic. J Telemed Telecare. 2002;8(2):102–106. doi: 10.1258/1357633021937550. http://jtt.rsmjournals.com/cgi/doi/10.1258/1357633021937550. [DOI] [PubMed] [Google Scholar]
- 23.Nijland N, Gemert-Pijnen J E W C, Kelders S M, Brandenburg B J, Seydel E R. Evaluation of the use of an "ask-the-expert" e-consultation service for support on health-related requests. Stud Health Technol Inform. 2010;160(Pt 2):821–825. [PubMed] [Google Scholar]
- 24.Greenhalgh Trisha, Robert Glenn, Macfarlane Fraser, Bate Paul, Kyriakidou Olivia. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82(4):581–629. doi: 10.1111/j.0887-378X.2004.00325.x. http://www.scholaruniverse.com/ncbi-linkout?id=15595944. [DOI] [PMC free article] [PubMed] [Google Scholar]