Abstract
Background
Telehealth appears to be an ideal mechanism for assisting rural patients and doctors and medical students/registrars in accessing specialist services. Telehealth is the use of enhanced broadband technology to provide telemedicine and education over distance. It provides accessible support to rural primary care providers and medical educators. A telehealth consultation is where a patient at a general practice, with the assistance of the general practitioner or practice nurse, undertakes a consultation by videoconference with a specialist located elsewhere. Multiple benefits of telehealth consulting have been reported, particularly those relevant to rural patients and health care providers. However there is a paucity of research on the benefits of telehealth to medical education and learning.
Objective
This protocol explains in depth the process that will be undertaken by a collaborative group of universities and training providers in this unique project.
Methods
Training sessions in telehealth consulting will be provided for participating practices and students. The trial will then use telehealth consulting as a real-patient learning experience for students, general practitioner trainees, general practitioner preceptors, and trainees.
Results
Results will be available when the trial has been completed in 2015.
Conclusions
The protocol has been written to reflect the overarching premise that, by building virtual communities of practice with users of telehealth in medical education, a more sustainable and rigorous model can be developed. The Telehealth Skills Training and Implementation Project will implement and evaluate a theoretically driven model of Internet-facilitated medical education for vertically integrated, community-based learning environments
Keywords: telehealth, medical education, enhanced broadband
Introduction
Background
Ensuring an appropriately trained and resourced medical workforce for regional, rural, and remote areas is critical to enhancing the welfare of rural Australians, as it is for the welfare of rural and remote communities globally [1]. Australian medical schools frequently arrange for some (or all) of their medical students to undertake part of their training away from major metropolitan areas with the aim of producing practitioners capable of, and motivated to, practice in regional, rural, and remote locations [2]. There is evidence that this approach results in sound educational outcomes [3,4]. In addition, this approach—together with targeted recruitment of graduates from rural areas—is highly important in creating a sustainable rural medical workforce [5].
Telehealth and Rural Medical Workforce Sustainability
Both a cause and effect of an inadequate rural medical workforce is lack of timely access to specialist advice for onsite rural health practitioners and patients [6,7]. One potential strategy for assisting rural patients and doctors who would benefit from specialist access is telehealth. Telehealth has been defined as the “use of telecommunication techniques for the purpose of providing telemedicine, medical education, and health education over a distance” [8]. For the purposes of providing support to rural primary care providers and patients, a telehealth consultation is where a patient at a general practice, with the assistance of the general practitioner (GP) or practice nurse, undertakes a consultation by videoconference with a specialist located elsewhere [9]. While telehealth consulting is reported to be underused in Australia [10], there is evidence that telehealth consulting can result in improved access to specialized health services for rural patients [11,12], upskill of rural GPs [13], improved rural workforce retention [13], significant health gains [12], and financial savings [12]. The recent Australian government-funded “Connecting Health Services with the Future” initiative was designed to address this issue by providing Medicare rebates and financial incentives for video consultations with specialists for patients outside of major metropolitan areas [9], supported by telehealth consulting standards set by the Royal Australian College of General Practitioners [14] and Australian College of Rural and Remote Medicine [15].
However, until recently, technical restrictions including requirements for necessary bespoke hardware, video cameras, and integrated services digital network (ISDN) connectivity [16] have limited most applications of telehealth in Australia, limiting access and scalability [16]. For example, most professional quality video consulting applications, up until recently, have used fully featured Web-based medical video consulting software. These applications, including Skype, provide encryption for data transmission security and Medicare Australia standards level audit trails [17]. While there is evidence that exposure to telehealth consulting improves physician attitudes to its use [18], there are many reasons for the low rates of adoption, namely insufficient training as a barrier to the uptake and use of this technology [19]. Other barriers to adoption of telehealth have been identified including lack of adequate equipment, poor connectivity, lack of telehealth consulting skills, scheduling difficulties, inadequate reimbursement schemes, and patient reluctance [20-26]. Therefore, what is required is training for health professionals to optimize the video consultation care provided [27], supported networks of health professionals using the technology [27], stakeholder involvement in implementation, and well-developed business models.
Virtual Communities of Practice and Rural Medical Workforce Training
A further challenge to a sustainable rural workforce in Australia is securing sufficient clinical placement sites for the numbers of rural undergraduate, prevocational, and vocational trainees required by the health system [2]. A number of factors, including reduced hospital in-patient stay times, increased use of medical technology, expanding training numbers, and a recognition of the need for generalist medical exposure, have resulted in an increase in the demand for community clinical placements [28-30]. Of particular concern in Australia is the capacity of rural clinical schools to accommodate increased training numbers [2]. Thus, new paradigms of health care professional training are required as it is now not unusual for a general practice to host medical and nursing students, pre-vocational doctors in training, and vocational GP trainees.
To manage this number of learners without exhausting the clinician teachers (and while still providing quality medical care and educational training), innovative models of teaching are required. One such model is vertically integrated teaching where all levels of learner, from undergraduate to vocational, contribute to a learning environment [31]. One way of conceptualizing vertically integrated learning environments is as communities of practice (CoPs). CoPs can be defined as “groups of people who share a concern or a passion for something they do and learn how to do it better as they interact regularly” [32]. There is strong evidence from business literature that CoPs can be effective in workplace training with measureable improvements in outcomes [33]. Geographically dispersed learners may also be connected by information technology tools to form virtual communities of practice (VCoPs) [34]. In medical education, VCoPs have been shown to be perceived by GP trainees to be useful in overcoming the isolation that can accompany training in rural areas [35].
The Role of Internet-Based Solutions for Rural Medical Workforce Challenges
Telehealth consulting has been reported to facilitate peer communication and reduce isolation for rural and remote doctors [11]. It has also been demonstrated that rurally relevant continuing medical education is effective in increasing practitioners’ confidence to practice in a rural location, reducing isolation and increasing intention to continue to work in a rural area [36]. There is also evidence that VCoPs can be effective in fostering information sharing and reducing professional isolation in GP training [37]. With the above evidence in mind, the Telehealth Skills Training and Implementation Project (TSTIP), funded through the Australian government’s Broadband Enabled Education, Skills and Services (BEES) program, aims to evaluate the introduction of two broadband Internet-based interventions in rural medical training. The first is the training of telehealth skills in medical schools coupled with the use of telehealth consulting as a teaching medium for undergraduate medical students. The second is the use of broadband-enabled “virtual clinics” to support high-quality medical education and promote vertically integrated teaching and VCoPs in rural general practice.
To guide the development of virtual communities of practice, with a view to improving knowledge sharing and overcoming medical professional isolation, the Health VCoP Framework has been developed [34]. This framework has been used in previous studies on GP training and covers the seven steps to be considered when implementing a VCoP. The steps include (1) organizing facilitation, both administrative and professional, (2) ensuring an ongoing champion and support from stakeholders, (3) establishing clear goals, (4) involving a “broad church” of users, (5) establishing a supportive environment, (6) measuring activity and progress against goals, and providing feedback to participants and opportunities for benchmarking, and (7) considering technical factors such as ease of use and access, synchronous and asynchronous interactions, and considering community factors such as providing high-quality content, building trust online, self-selection of membership, and encouragement of active and passive users.
The Health VCoP Framework will be used to help inform the project’s activities and also provide a template for assessing the success (or otherwise) of the project in fostering CoPs and VCoPs among the project’s participants. While creation of CoPs and VCoPs are aims of the project, there are other educational, technical, and administrative components of the trial that require evaluation. To aid conceptualization of the use of the Health VCoP framework for the project’s implementation and evaluation, the project plan is discussed in entirety under the Health VCoP headings in the following sections, recognizing that several measures are not relevant to the framework.
Methods
Project Implementation
Health VCoP Step 1: Organize Facilitators and Moderators—Both Clinical and Administrative
Virtual Clinics
To support high-quality medical education and promote vertically integrated teaching in rural general practice, we will run a series of “virtual clinics”. This will require both administrative and clinical facilitation. Clinicians associated with the University of Wollongong will provide clinical teaching, facilitating a virtual clinic that is transmitted to participating metropolitan, regional, rural and remote training sites, and to individual students and doctors who are involved in the program either through their connections with one of the universities involved or through one of the two GP training organizations involved.
The virtual clinic topics will be chosen with the aim of addressing problematic clinical issues for practice-based consulting in regional and rural areas. These sessions will be interactive with students, GP trainees, and GP preceptors in training sites being able to communicate, ask questions, discuss, and deliberate on clinical issues online in real-time. A key feature of the pedagogy of the virtual clinics will be a focus on teaching clinical reasoning in each case [38]. Pre-session reading and activities will be provided online before each session, and after session discussion will be available using interactive Web 2.0 technology. This provides an environment for creating a VCoP involving the students, GP trainees, GP preceptors, and tutors. Sessions will be recorded and made available to learners online after the live session.
A minimum of eight structured virtual clinics will be run during the trial, connected to participating GP practices, individuals, or educational facilities in rural and regional areas. Administrative facilitation will be undertaken by the TSTIP project team. Educational facilities suitable for involvement include the Shoalhaven campus of the University of Wollongong, the Southern Highlands facility of University of Wollongong School of Medicine, and the University of New England campus Armidale, New South Wales.
Telehealth Real-Patient Learning
Training sessions in telehealth consulting will be provided for participating practices and students. The trial will then use telehealth consulting as a real-patient learning experience for students, GP trainees, GP preceptors, and trainees. This educational model focuses on the student or GP trainee actively engaging in the patient’s journey’ [39] through the health system by being involved in the telehealth referral workup, consultation, debrief, and follow-up with consenting patients. It is also intended to provide experiential patient-based learning [40] of the utility of telehealth consulting for reducing professional isolation [11] in problematic clinical cases. It is planned that cases will then form the basis for in-practice teaching and also as subjects for the students’ and GP trainees’ reflective clinical logs. The students involved will be in the senior years of their training (third and fourth year) and are already involved in “parallel consulting” where they see patients prior to (or with) the GP preceptor, take histories, perform physical examinations, formulate management plans with the GP, and assist the GP with procedures. Patient participants will supply consent for this participation in keeping with professional guidelines [14,41]. Student involvement in telehealth consults with consenting patients will be a natural extension of their roles within the GP training practices.
Health VCoP Step 2: Champion and Support
The TSTIP is a multicenter phased project. The development phase began on July 1, 2012. The implementation phase, the subject of this evaluation, began on August 1, 2013 and will be finalized by April 30, 2015.
The University of Wollongong General Practice Academic Unit (GPAU) dedicated TSTIP team will provide the project champion role, engaging and ensuring ongoing support from stakeholders partners including the project consortium, which consists of University of Wollongong,
University of New England, University of Newcastle, University of Notre Dame Australia (Sydney Campus), Deakin University, Coast City Country General Practice Training, and GP Synergy.
Health VCoP Step 3: Establish Clear Goals
The educational goals for the TSTIP are to use broadband Internet to facilitate (1) the acquisition of medical skills, knowledge, or attitudes that improve confidence to practice in less supported medical environments [36], and (2) the development of professional networks and communities of practice to reduce the sense of professional isolation in less supported medical environments [34,35].
Health VCoP Step 4: A Broad Church
The primary target audience for the trial is the cohort of medical students, post-graduate pre-vocational doctors in training, GP trainees, and GPs in the regional, rural, and remote catchments of the project consortium. Collaborators in the consortium include universities with medical students in rural placements and Regional Training Providers with GP trainees in approximately overlapping rural areas, predominately in New South Wales.
Health VCoP Step 5: Establishing a Supportive Environment
The activities of the trial are designed to foster supportive educational interactions. The strategies employed by the trial include ensuring encouraging and positive moderation of all sponsored educational sessions, moderation of online feedback during sessions, nesting learning where possible in the GP training practices where the students and registrars are placed, and engaging the local GP supervisors and regional academic leaders in the learning processes.
Health VCoP Step 6: Measuring Activity and Progress and Providing Feedback
Real-time feedback to participants regarding their responses to the clinical content of the virtual clinics is part of the interactive nature and an integral aspect of the learning environment. In addition, the evaluation data from both virtual clinics and from telehealth consultations involving students and registrars that are collected as the project progresses will be used to continually review the project activities and adjust them appropriately. The evaluation approach, methodology, and metrics are discussed below.
Project Evaluation Approach
The experiential learning intrinsic to general practice-based clinical placements involves a mix of professional modeling and skills application, including the domains of clinical reasoning [38], evidence-based decision making, management processes, and professionalism [28,42]. The educational goals for the project’s activities reflect these learning emphases. Complex attributes, such as professionalism, are not readily assessed in closed-response type evaluations [43,44]. Hence, the normal processes for evaluating the impact of educational activities (pre- and post-intervention testing) are not readily employed in this trial. Additionally, there are many facets of the trial, for example, its impact on practices, changes in practice systems, and procedures and scalability that require evaluation in addition to the educational outcomes. In order to undertake this complex and multifaceted evaluation, the authors have chosen to incorporate the principles of a Responsive Evaluation [45]. The approach broadens the evaluation to include a wide range of stakeholder issues. The goal of a responsive evaluation is “to enhance the understanding of a program from the…perspective of insiders” [46]. Thus, there is a need to identify all stakeholders and involve them in assisting to define the criteria for the evaluation. In this evaluation, this will include representatives of the funder through to practice staff and technical experts through to specialist consultants. Evaluation will commence at the inception of the project and will run in parallel with the trial activities. Participation in the evaluation is voluntary for the project participants.
A priori criteria, developed in conjunction with the funder during the project development phase include (1) the number of learners involved, (2) educational benefit of telehealth consulting and virtual clinics, (3) the development of professional learning networks (CoPs), (4) performance of information technology network components, (5) human-technology interactions, (6) cost implications, and (7) sustainability of telehealth real-patient learning and virtual clinics.
The results of the above assessments combined with focused interview data from all stakeholders will be used to evaluate whether telehealth (clinical and educational) is a sustainable teaching and learning activity for the participating institutional organizations and GP teaching practices.
Methodological Approach
Overview
The project evaluation seeks to understand the impact of technology-mediated educational interventions in the contexts of complex social and human systems. A multisite case study offers an appropriate methodology for this purpose with the ability to provide findings with high internal validity [47,48]. A multisite case study method permits the investigation of a contemporary phenomenon within its context, where boundaries between context and phenomenon may be blurred and a variety of evidence sources are required [47,48]. This approach has the advantage of facilitating in-depth evaluation even when a limited sample size would not permit statistical generalizability of quantitative results [47]. The data to be collected from various facets of the trial in order to synthesize the case studies are discussed below.
Telehealth Skills Training
The telehealth consultation training sessions will be evaluated using surveys for students, GP preceptors, staff, and specialists administered at the end of the training program.
Activity Metrics
Data will be collected on an ongoing basis to track the learning activity for the trial. This includes numbers of participants in each component of the trial, the number of learning activities, and participant numbers. In addition, Internet usage data will be collected concerning the numbers and frequency of participants accessing and interacting with online learning activities.
Pre-Trial Participants’ Views, Barriers, and Facilitators
Initially, semistructured interviews will be held with stakeholders (GP preceptors, GP trainees, students, specialists, patients, and staff) in order to assist the development of evaluation parameters. Semistructured, rather than structured interviews will be used to identify a range of views, experiences, barriers, and facilitators regarding telehealth-based medical education.
Telehealth Consultations
After selected telehealth consultations, the participants (GP preceptor, specialist, student, and patient) will be asked to undertake a structured survey that will seek their reflections on the individual session. To the extent possible, the surveys will be conducted straight after the consultation in order to minimize recall errors [49]. In addition to data on perceptions, performance, and satisfaction, students will also be asked to reflect on the learning experience provided through the consultation. During consultations, technical system data will be collected relating to the speed of the connection, quality of images, and network/hardware/software reliability.
Telehealth Virtual Clinics
Similarly, data from all virtual clinic sessions will be collected from the participants (GP trainee, GP preceptor, or student) who will be asked to undertake a structured survey that will seek their reflections on the individual session. Students will also be asked to reflect on the learning experience provided through the virtual clinic session. Metrics concerning transmission speeds, numbers of users, interactions, and reliability of transmission will also be assessed. A mid-point analysis of collected data will be undertaken and adjustments to the trial introduced if required.
Post-Trial Participants’ Views and Evaluations
At the end of the trial period, semistructured interviews will be conducted with the GP preceptors, trainees, specialists, students, practice staff, IT support, patients, and Graduate School of Medicine (GSM) staff to obtain their overall impressions of telehealth consulting, performance of the network components, and the effectiveness of the training program. The process of inter-institution collaboration will be explored through stakeholder interviews and review of artefacts such as meeting minutes and project documents.
Cost Implications
Data will be collected regarding cost implications for the GSM through financial record review. The process of the inter-institution collaboration will be explored through stakeholder interviews and review of artefacts such as project documents. Outcomes for patients (health and cost-related) and doctors/practices (professional and business) will be assessed using a variety of sources including interviews and practice (not patient) records. Informal data collection will be undertaken throughout the trial concerning participants’ responses, concerns, successes, and challenges.
Case Study Recruitment and Data Collection
Table 1 outlines the proposed sites and estimated number of learners for the virtual clinic component of the case study. Table 2 outlines the proposed sites and estimated number of learners for the telehealth real-patient learning component of the trial. Table 3 summarizes the data to be collected and the analytical approach to the sets of data.
Table 1.
Proposed sites and learners for virtual clinic participation.
| Site type | Locations | Number of practices/ sites | Number of learners (estimated) |
| GP practice | Armidale | 3 practices | 28 |
| Illawarra | 8 practices | ||
| Shoalhaven | 1 practice | ||
| Mudgee | 1 practice | ||
| Grafton | 1 practice | ||
| Southern Highlands | 1 practice | ||
| Education center | Shoalhaven; Southern Highlands University of Wollongong | 2 sites | 50 |
| Geelong (Deakin) | 1 site | ||
| Sydney (University of Notre Dame Australia) | 1 site | ||
| Armidale (University of New England) | 1 site |
Table 2.
Proposed number of sites and learners for the telehealth real patient learning participation in GP practice (these learners are also included in Table 1 for virtual clinics).
| Locations | Number of practices/ sites | Number of learners (estimated) |
| Armidale | 3 practices | 20 |
| Illawarra | 2 practices | |
| Shoalhaven | 2 practices | |
| Mudgee | 2 practices | |
| Grafton | 1 practice | |
| Southern Highlands | 1 practice |
Table 3.
Data collection schedule and analyses.
| Data collection time point | Activity | Data collected | Anticipated number of evaluation participants | Analysis method |
| Baseline, at start of active trial | Telehealth consulting training sessions for participating practices and students | Anonymous pre- and post-telehealth training session surveys for GP preceptors, practice staff, and students | GP preceptors n=8 | Descriptive analysis of quantitative data |
| Practice staff n=10 | ||||
| Specialists n=10 | ||||
| Baseline interviews | Coded, re-identifiable, audio-recorded, and transcribed semistructured interviews with GP preceptors, GP trainees, students, specialists, patients, and staff to identify a wide range of views, experiences, barriers, and facilitators regarding telehealth-based medical education | GP preceptors n=8 | Thematic analysis of interview transcripts | |
| GP trainees n=6 | ||||
| Students n=20 | ||||
| Specialists n=8 | ||||
| Patients n=20 | ||||
| Practice staff n=10 | ||||
| During trial | Telehealth consultations as a “real-patient” learning modality | Structured, site-coded, anonymous surveys after 24 telehealth consultations over the duration of the project seeking views of the participants (GP preceptor and/or GP trainee, specialist, student) concerning the individual sessions | GP preceptors n=10 | Descriptive analysis of quantitative data |
| GP trainees n=6 | ||||
| Students n=20 | ||||
| Specialists n=12 | ||||
| Telehealth consultations as a “real-patient” learning modality | During the evaluated 24 telehealth consultations aggregated, anonymous technical system data will be collected relating to the speed of the connection and network/hardware/software reliability | Technical data from 24 telehealth consultations | Descriptive analysis of quantitative data | |
| Virtual clinics as a learning modality | Structured, site-coded, anonymous surveys after all telehealth virtual clinic sessions over the duration of the project (12 in total) seeking views of the participants (GP preceptor and/or GP trainee, specialist, student) concerning the individual sessions | GP preceptors n=10 | Descriptive analysis of quantitative data | |
| GP trainees n=6 | ||||
| Students n=70 | ||||
| Specialists n=4 | ||||
| Virtual clinics as learning modality | During the evaluated 12 telehealth virtual clinics, anonymous, aggregated technical system data will be collected relating to the speed of the connection, network/hardware/software reliability and participant interaction | Technical data from 12 telehealth virtual clinics | Descriptive analysis of quantitative data | |
| All activities | Data will be collected on an ongoing basis to track the learning activity for the trial. This includes numbers of participants in each component of the trial, the number of learning activities, and participant numbers | Coded, re-identifiable data from project records | Descriptive analysis of quantitative data | |
| Virtual clinic online interactive activities (post-session) | Aggregated, anonymous Internet usage data will be collected concerning the numbers and frequency of participants accessing and interacting with online learning activities relating to the project (pre- and post-virtual clinic online interaction) | Anonymous data from Internet Web-site usage statistics | Descriptive analysis of quantitative data | |
| All activities | Coded, re-identifiable informal data collection will be undertaken throughout the trial concerning participants’ responses, concerns, successes, and challenges | Coded, re-identifiable data from researcher field notes | Thematic analysis of written field notes | |
| Follow-up post-trial | Post-trial interviews | Coded, re-identifiable, audio-recorded, and transcribed semistructured interviews with the GP preceptors, GP trainees, specialists, students, patients, practice staff, IT support, and university staff to obtain their overall impressions of the project including the performance of the project components, the effectiveness, and acceptability of the training program and cost implications | GP preceptors n=8 | Thematic analysis of interview transcripts |
| GP trainees n=6 | ||||
| Students n=20 | ||||
| Specialists n=8 | ||||
| Patients n=8 | ||||
| Practice staff n=10 | ||||
| IT support n=4 | ||||
| University staff n=6 | ||||
| Post-trial evaluation | The process of the inter-institution collaboration will also be explored through review of artefacts such as project documents | Coded, re-identifiable data from project records | Thematic analysis of project artefacts | |
| Post-trial evaluation | Data will be collected regarding cost implications for the GSM through financial record review. Outcomes for patients (health and cost-related) and doctors/practices (professional and business) will be assessed using a variety of sources including interview data, practice records, and Medicare data | Coded, re-identifiable data from interviews and project participants’ records | Descriptive analysis of quantitative data | |
| Universities n=5 | ||||
| Practices n=6 | Thematic analysis of interview transcripts and records | |||
| Patients n=8 | ||||
| Post-trial evaluation | All data sources will be synthesized and analyzed in a multicenter case study approach to provide an in-depth evaluation of the trial, assess its sustainability, and guide future implementation | All data sources | Mixed-methods multisite case study analysis |
Descriptive analysis of quantitative data will be undertaken. Qualitative analyses will include content analysis and thematic analysis along established lines [50]. The Health VCoP Framework will be used as the theoretical framework for the qualitative analyses, as the development of communities of practice is a key hypothesized outcome of the trial [34]. All of the data sources will be synthesized and analyzed in a multicenter case study [47,48] to provide an in-depth evaluation of the trial, assess its sustainability, and guide future implementation.
Health VCoP Step 7: Consideration of Technical Factors
The trial has focused resources on developing user-friendly, accessible technology with an in-depth technical needs assessment of participants prior to the trial, expert information technology membership in the project executive committee, a technical subcommittee dedicated to advising on technical matters, and constant user feedback concerning the usability of the technology through session evaluation reports. The technical aspects of the project will be adjusted and refined in response to feedback and the progress of the trial. High-quality content of educational sessions will be maintained by engaging content experts across the speciality fields addressed.
Ethical Considerations
Ethics approval for the project has been obtained through the University of Wollongong Human Research Ethics Committee (reference HE13/238).
Free and Informed Consent
The purpose of the evaluation will be explained to participants by researcher or project officer at each relevant data collection point, including the fact that all responses will be coded and de-identified and that participation is completely voluntary.
Confidentiality of Participants’ Responses
Evaluation surveys will be anonymous. All interviews be transcribed, coded, and de-identified by a research assistant. Practice location data will not be reported, and comments reported in aggregate terms. For qualitative data, participants will be assigned a code that will link their de-identified data to a confidential and secure master sheet that will contain participants’ demographic details and location.
Ensuring Participants Can Withdraw From the Trial Without Detriment
All potential participants will be advised that their participation is voluntary and that they may refuse to participate or may withdraw from the study at any time without penalty.
Data Security
All information will be securely stored and accessible only by the project investigators.
Results
Results will be available when the trial has been completed in 2015.
Discussion
The TSTIP will implement and evaluate a theoretically driven model of Internet-facilitated medical education for vertically integrated, community-based learning environments. This is a pragmatic trial in working practices, with an evaluation method designed to capture the reality of outcomes, sustainability, and scalability of the project activities. Limitations of the project evaluation include a lack of data on formally assessable educational outcomes, an absence of controls, and a sample size inadequate for statistical generalizability of quantitative results. However, the evaluation should provide detailed, highly internally valid data. The results will not only inform the project’s expansion, but also be of value in informing similar initiatives elsewhere, with the goal of improving the sustainability of medical workforces and health care in rural and remote regions.
Abbreviations
- CoP
communities of practice
- GP
general practitioner
- GSM
Graduate School of Medicine
- TSTIP
Telehealth Skills Training and Implementation Project
- VCoP
virtual communities of practice
Footnotes
Conflicts of Interest: None declared.
References
- 1.Strasser R, Neusy AJ. Context counts: training health workers in and for rural and remote areas. Bull World Health Organ. 2010 Oct 1;88(10):777–82. doi: 10.2471/BLT.09.072462. http://europepmc.org/abstract/MED/20931063. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Eley DS, Young L, Wilkinson D, Chater AB, Baker PG. Coping with increasing numbers of medical students in rural clinical schools: options and opportunities. Med J Aust. 2008 Jun 2;188(11):669–71. doi: 10.5694/j.1326-5377.2008.tb01828.x. [DOI] [PubMed] [Google Scholar]
- 3.Worley P, Esterman A, Prideaux D. Cohort study of examination performance of undergraduate medical students learning in community settings. BMJ. 2004 Jan 24;328(7433):207–9. doi: 10.1136/bmj.328.7433.207. http://europepmc.org/abstract/MED/14739189. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Birden HH, Wilson I. Rural placements are effective for teaching medicine in Australia: evaluation of a cohort of students studying in rural placements. Rural Remote Health. 2012;12:2167. http://www.rrh.org.au/articles/subviewnew.asp?ArticleID=2167. [PubMed] [Google Scholar]
- 5.Henry JA, Edwards BJ, Crotty B. Why do medical graduates choose rural careers? Rural Remote Health. 2009;9(1):1083. http://www.rrh.org.au/articles/subviewnew.asp?ArticleID=1083. [PubMed] [Google Scholar]
- 6.Alexander C, Fraser J. General practitioners' management of patients with mental health conditions: the views of general practitioners working in rural north-western New South Wales. Aust J Rural Health. 2008 Dec;16(6):363–9. doi: 10.1111/j.1440-1584.2008.01017.x. [DOI] [PubMed] [Google Scholar]
- 7.Ellis IK, Philip T. Improving the skills of rural and remote generalists to manage mental health emergencies. Rural Remote Health. 2010;10(3):1503. http://www.rrh.org.au/articles/subviewnew.asp?ArticleID=1503. [PubMed] [Google Scholar]
- 8.ISO . Interoperability of telehealth systems and networks Part 2: Real-time sytems. International Organization for Standardization; 2004. [2014-12-15]. http://www.iso.org/obp/ui/ [Google Scholar]
- 9.DOHA . Specialist video consultations under Medicare. Canberra Australian Government; 2012. [2014-12-17]. http://www.mbsonline.gov.au/internet/mbsonline/publishing.nsf/Content/connectinghealthservices-Program%20Overview. [Google Scholar]
- 10.Smith AC, Gray LC. Telemedicine across the ages. Med J Aust. 2009 Jan 5;190(1):15–9. doi: 10.5694/j.1326-5377.2009.tb02255.x. [DOI] [PubMed] [Google Scholar]
- 11.Gagnon MP, Duplantie J, Fortin JP, Landry R. Implementing telehealth to support medical practice in rural/remote regions: what are the conditions for success? Implement Sci. 2006;1:18. doi: 10.1186/1748-5908-1-18. http://www.implementationscience.com/content/1//18. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Access Economics Financial and externality impacts of high-speed broadband for telehealth. 2010. [2014-12-17]. http://trove.nla.gov.au/work/81015874?selectedversion=NBD46895251.
- 13.Moffatt JJ, Eley DS. The reported benefits of telehealth for rural Australians. Aust Health Rev. 2010 Aug;34(3):276–81. doi: 10.1071/AH09794. [DOI] [PubMed] [Google Scholar]
- 14.RACGP Telehealth Standards Melbourne. 2012. [2014-12-17]. http://www.racgp.org.au/standards/telehealth.
- 15.ACRRM Telehealth standards. 2011. [2014-12-15]. http://www.ehealth.acrrm.org.au/telehealth-standards.
- 16.Gray LC, Smith AC, Armfield NR, Travers C, Croll P, Caffery LJ. Brisbane: UniQuest University of Queensland. 2011. [2014-12-16]. Telehealth Assessment Final Report http://www.mbsonline.gov.au/internet/mbsonline/publishing.nsf/Content/E9F2448C7C016735CA257CD20004A3AE/$File/UniQuest%20Telehealth%20Assessment%20Report%20.pdf.
- 17.About ConsultDirect 2012. [2014-12-15]. https://consultdirect.com.au/about.php.
- 18.Hanson D, Calhoun J, Smith D. Changes in provider attitudes toward telemedicine. Telemed J E Health. 2009 Jan;15(1):39–43. doi: 10.1089/tmj.2008.0052. [DOI] [PubMed] [Google Scholar]
- 19.Flynn D, Gregory P, Makki H, Gabbay M. Expectations and experiences of eHealth in primary care: a qualitative practice-based investigation. Int J Med Inform. 2009 Sep;78(9):588–604. doi: 10.1016/j.ijmedinf.2009.03.008. [DOI] [PubMed] [Google Scholar]
- 20.Larsen F, Gjerdrum E, Obstfelder A, Lundvoll L. Implementing telemedicine services in northern Norway: barriers and facilitators. J Telemed Telecare. 2003;9 Suppl 1:S17–8. doi: 10.1258/135763303322196196. [DOI] [PubMed] [Google Scholar]
- 21.Hu PJH, Chau PYK, Sheng ORL. Adoption of telemedicine technology by health care organizations: An exploratory study. J Org Comp Elect Com. 2002;12(3):197–221. [Google Scholar]
- 22.Helitzer D, Heath D, Maltrud K, Sullivan E, Alverson D. Assessing or predicting adoption of telehealth using the diffusion of innovations theory: a practical example from a rural program in New Mexico. Telemed J E Health. 2003;9(2):179–87. doi: 10.1089/153056203766437516. [DOI] [PubMed] [Google Scholar]
- 23.Barton PL, Brega AG, Devore PA, Mueller K, Paulich MJ, Floersch NR, Goodrich GK, Talkington SG, Bontrager J, Grigsby B, Hrincevich C, Neal S, Loker JL, Araya TM, Bennett RE, Krohn N, Grigsby J. Specialist physicians' knowledge and beliefs about telemedicine: a comparison of users and nonusers of the technology. Telemed J E Health. 2007 Oct;13(5):487–99. doi: 10.1089/tmj.2006.0091. [DOI] [PubMed] [Google Scholar]
- 24.Campbell JD, Harris KD, Hodge R. Introducing telemedicine technology to rural physicians and settings. J Fam Pract. 2001 May;50(5):419–24. [PubMed] [Google Scholar]
- 25.Hopp F, Whitten P, Subramanian U, Woodbridge P, Mackert M, Lowery J. Perspectives from the Veterans Health Administration about opportunities and barriers in telemedicine. J Telemed Telecare. 2006;12(8):404–9. doi: 10.1258/135763306779378717. [DOI] [PubMed] [Google Scholar]
- 26.Stanberry B. Telemedicine: barriers and opportunities in the 21st century. J Intern Med. 2000 Jun;247(6):615–28. doi: 10.1046/j.1365-2796.2000.00699.x. [DOI] [PubMed] [Google Scholar]
- 27.Gray LC, Smith AC, Armfield NR, Croll P, Caffery LC. Brisbane: UniQuest University of Queensland. 2011. [2014-12-17]. Telehealth Business Case, Advice and Options http://www.mbsonline.gov.au/internet/mbsonline/publishing.nsf/Content/E9F2448C7C016735CA257CD20004A3AE/$File/UniQuest%20Telehealth%20Business%20Case%20Advice%20and%20Options.pdf.
- 28.Norris TE, Schaad DC, DeWitt D, Ogur B, Hunt DD, Consortium of Longitudinal Integrated Clerkships Longitudinal integrated clerkships for medical students: an innovation adopted by medical schools in Australia, Canada, South Africa, and the United States. Acad Med. 2009 Jul;84(7):902–7. doi: 10.1097/ACM.0b013e3181a85776. [DOI] [PubMed] [Google Scholar]
- 29.Parry J, Greenfield S. Community-based teaching: killing the goose that laid the golden egg? Med Educ. 2001 Aug;35(8):722–3. doi: 10.1046/j.1365-2923.2001.01009.x. [DOI] [PubMed] [Google Scholar]
- 30.Thistlethwaite JE, Kidd MR, Hudson JN. General practice: a leading provider of medical student education in the 21st century? Med J Aust. 2007 Jul 16;187(2):124–8. doi: 10.5694/j.1326-5377.2007.tb01160.x. [DOI] [PubMed] [Google Scholar]
- 31.Thomson JS, Anderson KJ, Mara PR, Stevenson AD. Supervision--growing and building a sustainable general practice supervisor system. Med J Aust. 2011 Jun 6;194(11):S101–4. doi: 10.5694/j.1326-5377.2011.tb03139.x. [DOI] [PubMed] [Google Scholar]
- 32.Wenger E. Communities of practice: a brief introduction 2011. [2014-12-15]. https://scholarsbank.uoregon.edu/xmlui/bitstream/handle/1794/11736/A%20brief%20introduction%20to%20CoP.pdf?sequence=1.
- 33.Probst G, Borzillo S. Why communities of practice succeed and why they fail. European Management Journal. 2008;26(5):335–47. [Google Scholar]
- 34.Barnett S, Jones SC, Bennett S, Iverson D, Bonney A. General practice training and virtual communities of practice - a review of the literature. BMC Fam Pract. 2012;13:87. doi: 10.1186/1471-2296-13-87. http://www.biomedcentral.com/1471-2296/13/87. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Barnett S, Jones SC, Bennett S, Iverson D, Bonney A. Usefulness of a virtual community of practice and web 2.0 tools for general practice training: experiences and expectations of general practitioner registrars and supervisors. Aust J Prim Health. 2013;19(4):292–6. doi: 10.1071/PY13024. [DOI] [PubMed] [Google Scholar]
- 36.White CD, Willett K, Mitchell C, Constantine S. Making a difference: education and training retains and supports rural and remote doctors in Queensland. Rural Remote Health. 2007;7(2):700. http://www.rrh.org.au/articles/subviewnew.asp?ArticleID=700. [PubMed] [Google Scholar]
- 37.Barnett S, Jones SC, Caton T, Iverson D, Bennett S, Robinson L. Implementing a virtual community of practice for family physician training: a mixed-methods case study. J Med Internet Res. 2014;16(3):e83. doi: 10.2196/jmir.3083. http://www.jmir.org/2014/3/e83/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Linn A, Khaw C, Kildea H, Tonkin A. Clinical reasoning - a guide to improving teaching and practice. Aust Fam Physician. 2012;41(1-2):18–20. http://www.racgp.org.au/afp/201201/45593. [PubMed] [Google Scholar]
- 39.Maughan TS, Finlay IG, Webster DJ. Portfolio learning with cancer patients: an integrated module in undergraduate medical education. Clin Oncol (R Coll Radiol) 2001;13(1):44–9. doi: 10.1053/clon.2001.9213. [DOI] [PubMed] [Google Scholar]
- 40.Doshi M, Brown N. Whys and hows of patient-based teaching. Advances in Psychiatric Treatment. 2005;11(3):223–31. [Google Scholar]
- 41.ACRRM Telehealth Guidelines 2012. [2014-12-15]. http://www.ehealth.acrrm.org.au/system/files/private/ACRRM%20Telehealth%20Guidelines_v1.0_20120827.pdf.
- 42.Kilminster S, Cottrell D, Grant J, Jolly B. AMEE Guide No. 27: Effective educational and clinical supervision. Med Teach. 2007 Feb;29(1):2–19. doi: 10.1080/01421590701210907. [DOI] [PubMed] [Google Scholar]
- 43.Lynch DC, Surdyk PM, Eiser AR. Assessing professionalism: a review of the literature. Med Teach. 2004 Jun;26(4):366–73. doi: 10.1080/01421590410001696434. [DOI] [PubMed] [Google Scholar]
- 44.Spring B. Health decision making: lynchpin of evidence-based practice. Med Decis Making. 2008;28(6):866–74. doi: 10.1177/0272989X08326146. http://europepmc.org/abstract/MED/19015288. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Curran V, Christopher J, Lemire F, Collins A, Barrett B. Application of a responsive evaluation approach in medical education. Med Educ. 2003 Mar;37(3):256–66. doi: 10.1046/j.1365-2923.2003.01442.x. [DOI] [PubMed] [Google Scholar]
- 46.Abma TA. The Practice and Politics of Responsive Evaluation. American Journal of Evaluation. 2006 Mar 01;27(1):31–43. doi: 10.1177/1098214005283189. [DOI] [Google Scholar]
- 47.Chetty S. The Case Study Method for Research in Small-and Medium-Sized Firms. International Small Business Journal. 1996;15(1):73–85. [Google Scholar]
- 48.Gagnon Y-C. The case study as research method. A Practical Handbook. Quebec: Presses de l'Universite du Quebec; 2010. [Google Scholar]
- 49.Barrett LF, Barrett DJ. An Introduction to Computerized Experience Sampling in Psychology. Social Science Computer Review. 2001 May 01;19(2):175–185. doi: 10.1177/089443930101900204. [DOI] [Google Scholar]
- 50.Huston P, Rowan M. Qualitative studies. Their role in medical research. Can Fam Physician. 1998 Nov;44:2453–8. http://europepmc.org/abstract/MED/9839063. [PMC free article] [PubMed] [Google Scholar]