Abstract
PURPOSE
Electronic consultation (eConsult) services can improve access to specialist advice. Little is known, however, about whether and how often primary care clinicians adhere to the advice they receive. We evaluated how primary care clinicians use recommendations conveyed by specialists via the Champlain BASE (Building Access to Specialists through eConsultation) eConsult service and how eConsult affects clinical management of patients in primary care.
METHODS
This is a descriptive analysis based on a retrospective chart audit of 291 eConsults done between January 20, 2017 and August 31, 2017 at the Bruyère Family Health Team, located in Ottawa, Canada. Patients’ charts were reviewed until 6 months after specialist response for the following main outcomes: implementation of specialist advice by primary care clinicians, communication of the results to the patients, method, and time frame of communication.
RESULTS
Primary care clinicians adhered to specialist advice in 82% of cases. Adherence ranged from 62% to 93% across recommendation categories. Questions asked by primary care clinicians related to diagnosis (63%), management (27%), drug treatment (10%), and procedures (1%). Recommendations of the eConsult were communicated to patients in 79% of cases, most often by face-to-face visit (38%), telephone call (32%), or use of the patient portal (9%). Communication occurred in a median of 5 days.
CONCLUSIONS
We found little evidence of barriers to implementing specialist advice with use of eConsult, which suggests recommendations given through service were actionable. With a high primary care clinician adherence to specialist recommendations and primary care clinician–to-patient communication, we conclude that eConsult delivers good-quality care and improves patient management.
Key words: access to care, eConsult, primary care clinician adherence, professional practice, coordination of care, delivery of health care, integrated, communication, decision making, health informatics, practice-based research
INTRODUCTION
Access to specialist care remains a serious problem in Canada because of excessive wait times. The Commonwealth Fund’s 2016 survey reported that 56% of Canadian adults waited more than 4 weeks to see a specialist, placing the country last among participant countries, which averaged 36%.1 Although specialty referral volumes and wait times are increasing, the current referral process itself remains unsafe and inefficient in terms of timeliness of referrals, specialist access, and communication between primary and specialty care, all of which have an impact on patient safety.
Worldwide, different online consultation platforms are being used to improve access to specialist care and offer potential advantages over the usual process of a face-to face visit to the specialist, including rapid access to advice, improved communication, and educational benefits.2–7 Electronic consultation (eConsult) is an asynchronous communication tool that facilitates virtual conversation between clinicians about the patient. Through eConsult, primary care clinicians are able to manage some conditions with remote support by the specialist without patients themselves needing a face-to-face specialist appointment.
Although ample evidence has demonstrated eConsult’s ability to improve access to specialist advice, less is known about whether and to what extent primary care clinicians adhere to the advice they receive. Understanding of these measures is important, as it is up to the primary care clinician’s discretion whether they implement the advice they receive through eConsult; consequently, their action in response to advice given affects the care provided. To date, 3 US studies have been conducted on this issue, finding primary care clinician compliance with specialist recommendations ranging from 65% to 86%.8–10 As referrals lacking a clear consultative question and relevant clinical data often have been criticized for rendering a specialist unable to make a clear diagnosis or formulate a fully developed management plan,10 and eConsult has been shown to improve the consistency and clarity of communication from the referring primary care clinician,11 it is reasonable to expect that eConsult can facilitate adherence to recommendations by improving communication of primary care to specialty care.
In this study, we evaluated eConsult’s effect on clinical management of patients by measuring primary care clinicians’ adherence to specialist recommendations after completing an eConsult, and by determining whether and how primary care clinicians communicate results to patients. The results of this study will help address the existing gaps in the literature concerning electronic consultation and will add new insights on the value of the eConsult service. This information will be highly relevant not only for clinicians and patients, but also for policy makers.
METHODS
Design and Approval
We conducted a retrospective chart audit for patients whose primary care clinicians completed an eConsult on their behalf between January 20, 2017, and August 31, 2017. This study was approved by the Ottawa Health Sciences Research Ethics Board (no. 20170918-01H) and the Bruyère Continuing Care Research Ethics Board (M16-18-008).
eConsult Service
The Champlain BASE (Building Access to Specialists through eConsultation) eConsult service was developed in 2010 to provide faster access to specialist care and ease communication between primary care clinicians and specialists in Canada.5,12 Primary care clinicians with a nonurgent, patient-specific question can submit it to more than 100 specialty groups via a secure web-based platform. Additional patient information such as medical history, medication lists, and laboratory and imaging results can be added if needed. Specialists respond within 1 week and can provide primary care clinicians with recommendations, request additional information, or recommend a face-to-face referral. After the specialist responds, primary care clinicians and specialists can have ongoing communication until the primary care clinician closes the eConsult. Primary care clinicians are expected to communicate the results of the eConsult to the patient and implement any recommendations, if appropriate.
Setting
The study was conducted at the Bruyère Family Health Team (FHT), located in Ottawa, Canada. Family health teams are primary health care organizations that include a team of family physicians, nurse practitioners, registered nurses, social workers, dietitians, pharmacists, and other professionals who work together to provide primary health care for their community. The Bruyère FHT consists of 25 primary care clinicians, including 4 nurse practitioners, who provide care for more than 17,000 patients.
The clinic is a frequent user of the Champlain BASE eConsult service and has developed a unique way of integrating the eConsult service into daily practice. Primary care clinicians can create an eConsult using a form designed by the Bruyère FHT to emulate the standard eConsult form, which is integrated into the clinic’s electronic health record (EHR) system. This configuration allows primary care clinicians to request an eConsult by entering the patient data directly into the form without leaving the EHR. Once this step is done, the primary care clinician sends a message to the clinic’s referral clerks in the same manner as when requesting a traditional referral. The clerical team is trained to work with the eConsult service as a delegate. This means that the referral clerks log onto the eConsult server and upload all of the relevant data into the system. Once the specialist replies, the referral clerk retrieves the information, uploads it to the EHR, and notifies the primary care clinician. The referral clerks finalize the request, transfer the specialist response into the EHR, and notify the primary care clinician when the consultation is completed. The primary care clinician then completes all outstanding actions (eg, replying to the specialist if desired, responding to the mandatory close-out survey) in the EHR template, which the referral clerk replicates in the online platform. The entire management of the eConsult, including multiple cycles of communication in the context of the same case, occurs within the clinic EHR (with the primary care clinician taking the action within the EHR, and the referral staff incorporating that into the eConsult application on the web and then communicating these actions to the primary care clinician who originally requested the eConsult).
The patient portal is another online application that the clinic frequently uses. It allows patients and primary care clinicians to communicate online for nonurgent matters. Of the clinic’s 13,249 rostered patients, 25% have signed up for the patient portal.
Data Collection and EHR Review
We reviewed all eConsults between January 20, 2017, and August 31, 2017. Data collected from patients’ records related to the consulted specialty, questions asked by the primary care clinician, specialists’ recommendations, primary care clinicians’ actions linked to specialist advice, and method and date of communication of the results to the patient. We also collected demographic information for the patients and primary care clinicians, including sex, patient’s age, first 3 digits of patient’s postal code, and clinician type. No unique patient or clinician identifiers were collected. Date of specialist response was also recorded.
For each case, we selected the main question asked by the primary care clinician and the main specialist recommendation, and assigned each to 1 of 4 main classification groups: diagnosis, drug treatment, management, and procedure. Patient EHRs were reviewed for outcomes within 6 months after specialist response. We considered a recommendation followed if there was evidence of implementation at the primary care clinician level (eg, a laboratory requisition, prescription of new drug, documentation of discussing the advice, a referral letter, or documentation of reassurance). Recommendations were considered not implemented if there was evidence of or a possible reason noted in the chart as to why the primary care clinician did not follow the specialist advice, or both. “Not sure” was selected if there was no documentation at all about the reason or content of the eConsult.
We noted whether there was documentation of communication of the results to the patient (yes or no). “Not sure” was selected in cases where there was no clear evidence in the chart of the patient and primary care clinician discussing the results of the eConsult, yet they had a follow-up contact about symptoms similar to those listed in the original eConsult. We also noted the method of communication used by primary care clinicians, clerks, or nurses, and how many days after the specialist response the communication took place. We selected the method and date of the first attempt to inform the patient.
Interrater Reliability
All records were reviewed by a medical student (G.d.M.). As part of quality assurance for record abstraction, 1 principal investigator (C.L.) reviewed 6% of the abstracted data at 3 intermittent checkpoints.13 The randomly selected cases were compared on agreement regarding the classification of primary care clinician questions and specialist recommendations, and the adherence to specialist recommendations. Any disagreements were discussed in depth until agreement was reached and the remaining cases were abstracted accordingly. With the intermittent checkpoints, the percent agreement of primary care clinician questions increased from 60% to 100%, and the agreement of specialist recommendations and adherence increased from 60% to 88%. The κ statistic for primary care clinician adherence to specialist advice was 0.62. The principal investigator also conducted a secondary review of any cases that were flagged for verification.
Data Analysis
We performed descriptive analyses to examine patient and primary care clinician characteristics and estimate adherence to recommendations. The significance of categorical differences in adherence to specialist recommendations was determined by χ2 tests. We used percent agreement and κ statistics to assess interrater agreement between reviewers.
RESULTS
The Bruyère FHT requested 302 eConsults during the study period (Figure 1). After exclusions, a total of 291 eConsults were analyzed.
Figure 1.
Inclusion and exclusion of cases for analysis.
eConsult = electronic consultation; ED = emergency department.
Patient and Clinician Characteristics
Patient and primary care clinician characteristics are shown in Table 1. The most consulted specialties were dermatology (32%), orthopedics (8%), and neurology (7%) (Figure 2). The median specialist response time was 3 days, with 90% of eConsults answered within 7 days. Main questions asked by primary care clinicians were related to diagnosis in 63% of the cases. Other questions related to management (27%), drug treatment (10%), and procedures (1%). The distribution of primary care clinician questions subcategories and question examples per category are presented in Table 2.
Table 1.
Characteristics of Primary Care Clinicians and Patients (N = 291)
| Characteristic | Number (%) |
|---|---|
| Primary care clinicians | |
| Type | |
| Family physician | 255 (88) |
| Nurse practitioner | 36 (12) |
| Sex | |
| Female | 154 (53) |
| Male | 137 (47) |
| Patients | |
| Sex | |
| Female | 161 (55) |
| Male | 130 (44) |
| Age-group | |
| Child/adolescent (0-17 years) | 60 (21) |
| Adult (18-64 years) | 167 (57) |
| Older adult (≥65 years) | 64 (22) |
| Location | |
| Urban | 282 (97) |
| Rural | 5 (2) |
| Unidentified | 4 (1) |
Figure 2.
Distribution of specialty groups consulted for 291 completed eConsult cases
eConsult = electronic consultation; ENT = ear, nose, and throat; OB/GYN = obstetrics/gynecology.
Note: “Other” category contains specialties accounting for less than 2% of the eConsults: general surgery, radiology, genetics, vascular surgery, sports medicine, allergy & immunology, thoracic surgery, pain medicine, bariatric care, addictions, neurosurgery, and ophthalmology.
Table 2.
Types and Examples of Questions Asked by Primary Care Clinicians Via the eConsult Service
| Question Type | Questions, No. (%) | Question Example |
|---|---|---|
| Diagnosis | 182 (63) | |
| Interpretation of clinical findings | 104 (36) | My patient is a 17-year-old girl who has a fine vesicular rash since a week. Please assess this rash. Could this be a viral exanthema or scabies? (Pictures are attached.) |
| Interpretation of prior testing | 70 (24) | This patient has a mildly elevated prolactin level of 37. Macro prolactin testing is pending. If this elevation is not due to macro prolactin, is this level something to worry about? |
| Diagnostic work-up | 8 (3) | This 36-year-old and 24-weeks-pregnant female has a TSH of 3.7 and free T4 of 13. We are planning to order TPO antibodies; is there any other bloodwork you would recommend? |
| Management | 78 (27) | |
| General management | 70 (24) | I’m sending this eConsult regarding a patient with persistent rectal bleeding from internal hemorrhoids. I am wondering if you have suggestions to stop and pre- vent the bleeding. |
| Should I refer? | 8 (3) | I have a healthy 45-year-old male who describes issues with erectile dysfunction for the past 7 years. Because of his young age, I am unsure of the approach to take at this time. Would this be someone who should be seen by urology? |
| Drug treatment | 28 (10) | |
| Drug of choice for particular condition | 15 (5) | What would you recommend for treatment for hair loss related to PCOS? |
| Indications for initiating therapy | 7 (2) | Would this 79-year-old man benefit from anticoagulation with warfarin? |
| Change in dose or stopping drug | 4 (1) | This patient is requesting to come off her lithium, which she has been on for many years. Her mood has been stable for a long time. Can you advise on the best way to decrease and eventually stop the lithium? |
| Adverse effects/interactions | 2 (1) | Is it safe to start Celexa in a pregnant patient with a declining mood and history of depression? |
| Procedure | 3 (1) | This girl has a small mole that has been the same for a long time, but recently bled without any trauma. The lesion is not painful or pruritic. It is a well-demarcated, round, homogenous 2-mm mole on her arm. Given the bleeding, would you recommend a biopsy? |
eConsult = electronic consultation; PCOS = polycystic ovary syndrome; free T4 = free thyroxine; TSH = thyroid stimulating hormone; TPO = thyroid peroxidase.
Primary Care Clinician Action on Specialist Recommendations
The overall percentage of specialist recommendations implemented by the primary care clinician was 82%. In 12% of cases, there was no documentation in the medical record regarding the content of the eConsult, and in 6%, the primary care clinician did not implement the recommendation. We found that primary care clinicians did not implement specialist advice for multiple reasons, including resolution of the patient’s symptoms (9 cases), primary care clinician belief that another diagnosis was more likely (3 cases), patient’s prior use of the recommended drug (1 case), the eConsult came too late for patients’ request (1 case), the primary care clinician still wanted to pursue a plan of action for reassurance (2 cases), and the primary care clinician discussed the eConsult with a pharmacist, who did not agree with the specialists’ treatment suggestion (1 case).
Figure 3 shows the specialist recommendation categories and primary care clinician adherence per category. Adherence ranged from 62% to 93%. In our study sample, the most common recommendation was to start a new drug treatment (21%), and this recommendation was followed by primary care clinicians in 84% of cases. The category “management – self-management patient” was created for recommendations that were meant to be executed by patients themselves (eg, dietary modification, using particular shampoo, wearing sup portive shoes), whereas “management – by primary care clinician” included use of liquid nitrogen, orthostatic measurements, and sending new photos to the eConsult specialist.
Figure 3.
Recommendations given by specialists and adherence of primary care clinicians to the recommendations for 291 completed eConsult cases.
Watchful waiting/doing nothing was recommended in 11% of eConsults, and this recommendation was followed by primary care clinicians in 76% of these cases. In 19 cases, the specialists reassured the clinician that no further investigation (laboratory or imaging) was needed; in 9 cases, that no treatment was needed; in 3 cases, that no referral was needed; and in 2 cases, that no procedure was needed.
Clinician-Patient Communication After eConsult
We found evidence that the primary care clinicians communicated the eConsult results to patients in 79% of cases. No evidence was found in 12% of cases, while in 9% of cases, a follow-up was requested but the content of the discussion was not recorded.
Figure 4 shows the methods of communication. The largest share of patients were informed about the results via a face-to-face contact with the primary care clinician (38%). The median time to communication after specialist response was 5 days, with 90% of patients being contacted within 46 days. Furthermore, communication through a face-to-face visit took place for 50% of patients within 13 days. Meanwhile, 50% of the patients who were informed via a telephone call, voice mail, or the patient portal got their results within only 2 days.
Figure 4.
Communication of eConsult results to the patient by primary care clinicians and distribution of communication methods used.
eConsult = electronic consultation; PCC = primary care clinician.
aTelephone call by a clerk (2 cases), face-to-face contact with a nurse (1 case), unclear which method was used (2 cases).
Dermatology vs Nondermatology Cases
A considerable proportion of eConsults pertained to dermatology, so we compared the 92 dermatology cases with the 199 nondermatology cases on type of question and answer, as well as adherence to recommendations. We found most dermatology specialists gave recommendations about starting a new drug treatment (42%), whereas nondermatology specialists recommended other management (19%), ordering laboratory tests (14%), or watchful waiting/doing nothing (14%). Primary care clinician adherence to recommendations was 80% in dermatology cases vs 83% in nondermatology cases. In 12% of dermatology cases, however, the primary care clinician did not adhere to the specialist advice, compared with just 3% of nondermatology cases, a difference that was statistically significant by Bonferroni-corrected χ2 tests.14 The higher proportion of nonadherence in the dermatology vs nondermatology cases was due mainly to a large number of the former in which symptoms had resolved in the interim.
DISCUSSION
The majority of specialist advice provided through eConsult was actionable and acted on. Primary care clinicians seldom opted not to implement the recommendations, and we found very few barriers to implementing specialists’ advice. In addition, we found evidence of communication in a high percentage of cases, with primary care clinicians connecting with patients themselves in the majority of the cases through telephone calls, in-person visits, or electronic communication. To our knowledge, this is the first Canadian study to examine primary care clinician adherence to advice received through eConsult.
Our findings are consistent with research previously conducted in the United States. A study by Wrenn et al9 found an adherence rate of 65% to 86% for various types of recommendation. Cruz et al10 examined adherence to advice in endocrinology cases and found an overall implementation rate of 76%. Pecina et al8 found primary care clinician adherence in 78% of the cases, further noting that in 6% of cases, the patient declined the recommendation. Communication with patients in that study was documented in 87% of cases, most frequently by primary care clinicians via telephone (23%), patient portal (23%), mail (5%), and face-to-face visit (11%).8 This latter point contrasts with our findings, which showed a higher proportion of patient follow-up occurring face to face (38% vs 11% in that study). This difference could be due to differences in study population, health system organization, or both. Those authors conducted their study in a multispecialty group practice, whereas the Bruyère FHT is primarily staffed with family physicians and nurse practitioners, who are probably more accessible for a timely follow-up visit. Differences in remuneration could also be a factor, as clinicians may be more likely to follow-up in person vs over the telephone if they are paid only for the former.
eConsult differs from the traditional referral-consultation process by allowing primary care clinicians and specialists to share information and discuss a patient’s case before the primary care clinician decides whether to refer. Evidence suggests that patients value improved communication in their care, and wish to be kept abreast of the health care process.15 The eConsult service facilitates this inclusion by allowing primary care clinicians to retain a central role in more patients’ care. As a result, some primary care clinicians have expressed concern that eConsult is shifting work away from specialists and onto primary care clinicians, increasing their workload.16,17 Although our study did not assess primary care clinician workload, we found that they chose to inform patients of the eConsult’s results themselves in most cases—including through face-to-face appointments, the most commonly used medium. Furthermore, eConsult provided the added benefit of clarity for all physicians involved. In traditional referrals, gaps in communication are common. Specialists are often not informed about a patient’s medical history and reason for consultation, while primary care clinicians regularly describe receiving no information on the referral’s outcome.1,18,19 Occasionally, these breakdowns in communication can make it unclear which physician is responsible for the patient’s care,20 an issue that primary care clinicians and specialists alike describe as negatively affecting patient care.18,19 The eConsult service removes ambiguity by providing a forum for clear communication between clinicians, and retains a record of the complete exchange. Lastly, eConsult has demonstrated educational benefits, with primary care clinicians regularly describing how information they learned by completing an eConsult case can be applied to future patients, expanding their scope of practice.2 Such responses, coupled with reports of high or very high satisfaction in 93% of cases,2 suggest that most primary care clinicians find eConsult’s benefits outweigh any additional workload the service might entail.
In our study, specialists recommended against ordering laboratory tests or imaging, starting treatment, or referring in 11% of the cases. A global movement called Choosing Wisely was initiated because of the high proportion of potentially unnecessary tests, treatments, and procedures.21 As these unnecessary interventions can potentially harm the patient, Choosing Wisely encourages clinicians and patients to think about possible consequences of management steps and make judicious decisions. Our findings show that eConsult helps primary care clinicians and patients to choose wisely.
Our study has several limitations. First, the study population came from a single large academic urban health center, which limits the generalizability of our findings. Second, because of the retrospective design of our study and the limitations of medical record audits, which rely on information captured in patients’ records, it is possible that we missed some information not documented there. The fact that we used a single main abstractor was both a limitation and a strength. This approach enhanced consistency, although the quality of abstraction may differ based on the clinical experience of the abstractor. To ensure quality, we introduced a second reviewer having a high level of clinical experience to review the interpretation of the data during several checkpoints throughout the study. Any differences were discussed and resolved with the primary chart abstractor, and modifications to the interpretation were implemented accordingly.
In conclusion, the eConsult service delivers specialist advice that is actionable by primary care clinicians and communicated in a timely fashion to patients. Primary care clinicians adhered to the advice they received in 82% of cases and reached out to patients with the results in 79% of cases. Turnaround was quick, with primary care clinicians contacting patients within a median of 5 days. This high level of adherence suggests that primary care clinicians consider the advice they receive valuable and that eConsult contributes to the delivery of prompt, high-quality care. Replication of this study is warranted to research the use of eConsult in diverse primary practice settings across Canada. To read or post commentaries in response to this article, see it online at http://www.AnnFamMed.org/content/17/2/150.
Acknowledgments
We would like to thank the primary care clinicians and clinic staff at the Bruyère Family Health Team for supporting and using the service, and Justin Joschko for his assistance in editing the manuscript and preparing it for publication.
Footnotes
Conflicts of interest: authors report none.
Funding support: Funding for this project was provided by the Canadian Institutes of Health Research.
Disclaimer: The funder played no part in the study design, collection, analysis, or interpretation of the data; in the writing of the report; or in the decision to submit the article for publication.
Previous presentations: Presented at Trillium Primary Health Care Research Day; June 6, 2018; Toronto Ontario, Canada.
References
- 1.Canadian Institute for Health Information. How Canada compares: results from the Commonwealth Fund’s 2016 international health policy survey of adults in 11 countries. https://www.cihi.ca/sites/default/files/document/text-alternative-version-2016-cmwf-en-web.pdf Accessed Jun 12, 2018.
- 2.Liddy C, Afkham A, Drosinis P, Joschko J, Keely E. Impact of and satisfaction with a new eConsult service: a mixed methods study of primary care providers. J Am Board Fam Med. 2015; 28(3): 394–403. [DOI] [PubMed] [Google Scholar]
- 3.Liddy C, Drosinis P, Keely E. Electronic consultation systems: worldwide prevalence and their impact on patient care–a systematic review. Fam Pract. 2016; 33(3): 274–285. [DOI] [PubMed] [Google Scholar]
- 4.Vimalananda VG, Gupte G, Seraj SM, et al. Electronic consultations (e-consults) to improve access to specialty care: a systematic review and narrative synthesis. J Telemed Telecare. 2015; 21(6): 323–330. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Keely E, Liddy C, Afkham A. Utilization, benefits, and impact of an e-consultation service across diverse specialties and primary care providers. Telemed J E Health. 2013; 19(10): 733–738. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Rodriguez KL, Burkitt KH, Bayliss NK, et al. Veteran, primary care provider, and specialist satisfaction with electronic consultation. JMIR Med Inform. 2015; 3(1): e5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Keely EJ, Archibald D, Tuot DS, Lochnan H, Liddy C. Unique educational opportunities for PCPs and specialists arising from electronic consultation services. Acad Med. 2017; 92(1): 45–51. [DOI] [PubMed] [Google Scholar]
- 8.Pecina JL, Frank JM, North F. A retrospective study on how primary care providers manage specialists’ recommendations after an e-consultation. SAGE Open Med. 2016; 4: 2050312116682127. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Wrenn K, Catschegn S, Cruz M, Gleason N, Gonzales R. Analysis of an electronic consultation program at an academic medical centre: Primary care provider questions, specialist responses, and primary care provider actions. J Telemed Telecare. 2017; 23(2): 217–224. [DOI] [PubMed] [Google Scholar]
- 10.Cruz ML, Gleason N, Wang M, Wrenn K, Gonzales R. Transforming the endocrine consult: asynchronous provider consultations. Endocr Pract. 2015; 21(5): 514–521. [DOI] [PubMed] [Google Scholar]
- 11.Chen AH, Yee HF., Jr Improving the primary care-specialty care interface: getting from here to there. Arch Intern Med. 2009; 169(11): 1024–1026. [DOI] [PubMed] [Google Scholar]
- 12.Liddy C, Rowan MS, Afkham A, Maranger J, Keely E. Building access to specialist care through e-consultation. Open Med. 2013; 7(1): e1–e8. [PMC free article] [PubMed] [Google Scholar]
- 13.Liddy C, Wiens M, Hogg W. Methods to achieve high interrater reliability in data collection from primary care medical records. Ann Fam Med. 2011;9(1):57–62. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Sharpe D. Your chi-square test is statistically significant: Now what? Pract Assess Res Eval. 2015; 20(8): 1–10. [Google Scholar]
- 15.Anderson R, Barbara A, Feldman S. What patients want: a content analysis of key qualities that influence patient satisfaction. J Med Pract Manage. 2007; 22(5): 255–261. [PubMed] [Google Scholar]
- 16.Lee MS, Ray KN, Mehrotra A, Giboney P, Yee HF, Jr, Barnett ML. Primary care practitioners’ perceptions of electronic consult systems: a qualitative analysis. JAMA Intern Med. 2018; 178(6): 782–789. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Gleason N, Ackerman S, Shipman SA. eConsult–transforming primary care or exacerbating clinician burnout? JAMA Intern Med. 2018; 178(6): 790–791. [DOI] [PubMed] [Google Scholar]
- 18.O’Malley AS, Reschovsky JD. Referral and consultation communication between primary care and specialist physicians: finding common ground. Arch Intern Med. 2011; 171(1): 56–65. [DOI] [PubMed] [Google Scholar]
- 19.Stille CJ, McLaughlin TJ, Primack WA, Mazor KM, Wasserman RC. Determinants and impact of generalist-specialist communication about pediatric outpatient referrals. Pediatrics. 2006; 118(4): 1341–1349. [DOI] [PubMed] [Google Scholar]
- 20.Keely E, Tsang C, Liddy C, Farrell B, Power B, Way C. Rationale and model for integrating the pharmacist into the outpatient referral-consultation process. Can Fam Physician. 2016; 62(2): 111–114. [PMC free article] [PubMed] [Google Scholar]
- 21.Choosing Wisely Canada. Facts about unnecessary tests, treatments, and procedures. https://choosingwiselycanada.org/about/ Accessed Jun 12, 2018.