Abstract
Background
Technology-enhanced learning (TEL) programs are increasingly seen as the way in which education for health care professionals can be transformed, giving access to effective ongoing learning and training even where time or geographical barriers exist. Given the increasing emphasis on this mode of educational support for health care practitioners, it is vital that we can effectively evaluate and measure impact to ensure that TEL programs are effective and fit for purpose. This paper examines the current evidence base for the first time, in relation to the evaluation of TEL programs for health care professionals.
Objective
We conducted a systematic review of the current literature relating to the evaluation of TEL programs for health care professionals and critically appraised the quality of the studies.
Methods
This review employed specific search criteria to identify research studies that included evaluation of TEL for health care professionals. The databases searched included Medline Ovid, Cumulative Index of Nursing and Allied Health Literature Plus Advanced, Applied Social Sciences Index and Abstracts, ZETOC, Institute of Electrical and Electronics Engineers Explore Digital Library, Allied and Complementary Medicine, and Education Resources Information Center between January 2006 and January 2017. An additional hand search for relevant articles from reference lists was undertaken. Each of the studies identified was critically appraised for quality using the Crowe Critical Appraisal Tool. This approach produced a percentage total score for each study across specified categories. A proportion of the studies were independently assessed by an additional two reviewers.
Results
The review identified 21 studies that met the inclusion criteria. The studies included scored totals across eight categories within a range of 37%-95% and an average score of 68%. Studies that measured TEL using learner satisfaction surveys, or combined pretest and posttest knowledge score testing with learner satisfaction surveys, were found to be the most common types of TEL evaluations evident in the literature. The studies reviewed had low scores across reporting on ethical matters, design, and data collection categories.
Conclusions
There continues to be a need to develop effective and standard TEL evaluation tools, and good quality studies that describe effective evaluation of TEL education for health care professionals. Studies often fail to provide sufficient detail to support transferability or direct future TEL health care education programs.
Keywords: technology-enhanced learning, evaluation, e-learning, blended learning, digital learning, program evaluation, effectiveness
Introduction
The term technology-enhanced learning (TEL) is often used to describe a broad field of digital technologies used to support and mediate educational activities [1]. In this review, the term TEL is used to describe activities that are totally digitally mediated and those that are blended with more traditional educational approaches. The last two decades have seen considerable growth in the use of technology within higher education at undergraduate and postgraduate levels across the world [2]. Effective and ongoing continuous professional development (CPD) and education are essential to the delivery of high quality health and care services. TEL is increasingly presented as a means by which learners can be provided with enhanced or transformed educational experiences.
A range of published reports have highlighted TEL as an effective method to support health care education [3,4]. In their e-learning strategy, The Higher Education Funding Council England [5] summarized three levels of potential benefits of TEL: (1) e fficiency, whereby existing processes can be carried out in a more cost-effective, time-effective, sustainable, or scalable manner; (2) enhancement, which improves existing processes and outcomes; and (3) t ransformation, representing radical change in existing processes or the introduction of new processes. The recognition of the need for continuing education and effective work-based training to support health care professionals to deliver good quality, safe, and effective care is widely accepted [6,7]. The increasing demands for effective and affordable health care education in light of resource and time constraints, together with improved access to hardware, software, and the popularity of blended learning formats, means that TEL is increasingly considered an ideal approach within health care education.
The general availability of mobile and flexible technologies enables learners to minimize time away from health care settings to undertake training and to engage with learning resources when and where they are most suitable to their needs [8]. TEL offers a range of specific advantages for health care education, given the flexibility to update learning resources in a fast-changing field, and the scope offered for learners to share knowledge and learn critical clinical skills and decision making safely in nonclinical environments [9].
The ability to demonstrate the added value and impact of TEL for health care education remains challenging. Previous authors have captured the nature of the challenges in the review and evaluation of TEL within medical education [10-12]. Pickering and Joynes [13] highlighted the lack of robust evidence and meaningful evaluation to support widespread implementation of TEL resources. The main challenges concern a lack of clarity around the purpose of evaluation, comprehensiveness, depth, and methodology choice to support development of the required evidence base, upon which to build effective future TEL health care programs. We need reliable, practical mechanisms to evaluate: the value for money; equity of access; and learner, service, and organizational benefits that TEL may bring [14]. There is a need to critically examine the literature on TEL implementation and evaluation within health care to guide production and implementation of effective TEL health care education programs now and in the future.
A range of studies exist in the TEL literature which document implementation of TEL within medical and health care educational approaches. However, studies demonstrating a comprehensive TEL evaluation or use of standardized TEL evaluation tools in practice are fewer in number. Previous authors such as Ellaway [10,11], Cook and Ellaway [12], and Pickering and Joynes [13] (amongst others) have highlighted the need for robust evaluation, and have set out to develop both TEL educational standards and frameworks for evaluating TEL in medical education. Cook and Ellaway [12] have proposed a general model for evaluation. Pickering and Joynes [13] have proposed what they consider to be a more holistic TEL evaluation model for medical education.
This systematic review of the literature aimed to identify studies that have implemented TEL evaluation for CPD and postgraduate or work-based TEL health care education programs, and to assess these using a published critical appraisal tool. The studies identified provide an evidence base for the evaluation and development of future TEL programs for health care professionals.
Methods
Design
The review was carried out using a systematic integrative review method. This method allows for the inclusion of empirical and theoretical literature and quantitative and qualitative studies. This method enabled an increased number of studies to be included in the review and is appropriate for the review of evidence to highlight gaps in the literature [15].
Eligibility and Inclusion
The inclusion and exclusion criteria for this review were developed using the Population, Intervention, Comparison, and Outcomes (PICOS) [16] framework for systematic reviews, which is illustrated in Table 1. In devising the search strategy for this study, the PICOS framework has been used as a search tool and as an organizing framework to list terms by the main concepts in the search question. This framework is commonly used to identify components of clinical evidence for systematic reviews in evidence-based medicine and is endorsed by the Cochrane Collaboration [16].
Table 1.
Application of the Population, Intervention, Comparison, and Outcomes (PICOS) Framework to the research question. TEL: technology-enhanced learning.
| Parameter | Details |
| Participants | Health care professionals in full-time or part-time employment undertaking continuing professional development that is delivered using TEL (full time higher and further education students, school learners excluded). |
| Intervention | Studies using a TEL evaluation tool or framework to evaluate technology-enhanced health care education programs. The evaluation tool or framework must be used to evaluate a program for health care professionals. |
| Comparison | Some studies will have no comparison or comparator; others will examine one type of TEL approach against another. |
| Outcomes | Study must include: (1) evaluation of effective use of TEL, (2) the techniques being evaluated must be sufficiently specified, (3) assessment of learning outcomes, and (4) assessment of educational content. |
| Study design | Both empirical and theoretical research published in English between 2006-2017 from peer reviewed journals, conference papers. Systematic reviews and meta-analyses will not be included. Opinion papers will be excluded. |
Search Strategy
The following electronic databases were searched: Medline Ovid, Cumulative Index of Nursing and Allied Health Literature (CINAHL) Plus Advanced, Applied Social Sciences Index and Abstracts (ASSIA), ZETOC, Institute of Electrical and Electronics Engineers (IEEE) Explore Digital Library, Allied and Complementary Medicine (AMED), and Education Resources Information Center (ERIC) between January 2006 and January 2017 (see Multimedia Appendix 1 for details). The search was conducted using three concepts (and appropriate synonyms for each) across the selected databases: technology-enhanced learning, health care, and educational measurement. A total of 13 synonyms were used in the literature search. These phrases included technology-enhanced learning, technology-enhanced education, e-learning, e-education, blended learning, blended education, digital learning, digital education, evaluation studies, program evaluation, effectiveness, validation studies, and intervention. Reference lists were hand searched for relevant studies. The flow diagram in Figure 1 illustrates the search strategy.
Figure 1.
Literature search exclusion chart.
Data Extraction
A specific data extraction tool was developed based on the inclusion and exclusion criteria. Each article was reviewed and information was extracted in relation to participant type, study design, sample size, types of TEL used, TEL evaluation tool used, and key study findings.
Quality Assessment
Critical appraisal is a standardized way of assessing research so that decisions can be made based on the best evidence available [17]. The Crowe Critical Appraisal Tool (CCAT) was developed to provide a tool that can reliably assess a range of research designs, provide a comprehensive appraisal approach, and provide a suitable scoring method [17]. Quality assessment was conducted for all articles included within the study. The CCAT was used to assess quality across reporting items in eight categories: Preamble, Introduction, Design, Sampling, Data collection, Ethical matters, Results, and Discussion. The items were rated on a nominal scale (Present/Absent/Not applicable). The CCAT was selected as an appropriate critical appraisal tool for this study, as it can be reliably applied across a range of study designs and has been applied to a range of both quantitative and qualitative studies [17]. The CCAT emphasizes the importance of measuring and recording scores for each of the categories rather than simply the final score for each study. This approach prevents papers that score high overall, but poorly in one or more categories, from becoming less visible than papers that score highly throughout all categories. A subset of the studies [14,18-20] was randomly selected for scoring by additional reviewers (SM and HVW).
Results
Search Outcome
A total of 430 articles were identified in the initial titles search of the online databases and hand search of the literature. Following the application of the inclusion criteria to the titles and abstracts, 60 articles remained. The full text of the 60 articles were evaluated using the inclusion criteria and a total of 21 articles [14,18-37] were identified as suitable for the review.
Multimedia Appendix 2 shows a summary of the information extracted from the studies. Ten of the studies measured TEL using learner satisfaction surveys, 8 combined pretest and posttest knowledge score testing with learner satisfaction surveys (1 with pretest and posttest knowledge score testing only), and 2 used qualitative frameworks. The selected studies described the use of a variety of different TEL modes or combinations of each, including: 12 used Web-based e-learning, 4 used learning management systems, 2 used video simulation, and 6 used blended learning formats.
Quality Assessment
The results of the quality assessment are summarized within Table 2. The CCAT scores for the studies selected using the inclusion criteria indicate that the quality of studies varied greatly across the range of research parameters examined. Of the 21 papers, 19 presented sufficient information to be included in the CCAT evaluation. Two papers [14,18] were identified as TEL articles rather than research papers with CCAT scores in early sections too low to be viable to continue scoring, in accordance with CCAT user guidance. Ten studies scored above the average score of 68% overall and 9 scored less than this average (these are italicized within Table 2). The lowest average-scored sections for the set of 19 studies were in Ethical Matters, with a score of 2/5. Design, Data Collection, and Results each produced average scores of 3/5. Preliminaries, Discussion, Introduction, and Sampling sections each produced the higher average scores of 4/5 for the studies examined.
Table 2.
Crowe Critical Appraisal Tool (CCAT) scores summary. Italics indicate studies that scored less than the average score of 68%. N/A: not applicable.
| Authors | CCAT category | Raw score n (%) |
|||||||
| Preliminaries | Introduction | Design | Sampling | Data collection |
Ethical matters |
Results | Discussion | ||
| Akroyd et al [14] | 2 | 1 | N/A | N/A | N/A | N/A | N/A | N/A | 0 |
| Lotrecchiano et al [18] | 1 | 1 | N/A | N/A | N/A | N/A | N/A | N/A | 0 |
| Westbrook [37] | 2 | 4 | 1 | 1 | 3 | 0 | 1 | 3 | 15 (37.5) |
| Konstantinidis et al [31] | 4 | 3 | 1 | 2 | 1 | 0 | 1 | 4 | 16 (40) |
| Walsh et al [35] | 4 | 3 | 2 | 2 | 2 | 0 | 2 | 4 | 19 (47.5) |
| Heartfield et al [29] | 2 | 4 | 2 | 2 | 2 | 1 | 3 | 4 | 20 (50) |
| Chuo et al [24] | 3 | 4 | 3 | 3 | 4 | 0 | 2 | 3 | 21 (52.5) |
| Wang [36] | 2 | 2 | 3 | 2 | 2 | 5 | 4 | 2 | 22 (55) |
| Gill [27] | 3 | 4 | 4 | 2 | 3 | 2 | 3 | 4 | 23 (57.5) |
| Ingelbeen et al [30] | 4 | 5 | 3 | 4 | 1 | 2 | 2 | 4 | 25 (62.5) |
| Safwat and Pourabdollah [33] | 3 | 4 | 3 | 4 | 4 | 0 | 4 | 4 | 26 (65) |
| Goldberg Goetz et al [28] | 5 | 5 | 3 | 5 | 4 | 0 | 3 | 5 | 30 (75) |
| Byrne et al [22] | 3 | 5 | 4 | 4 | 4 | 4 | 4 | 4 | 32 (80) |
| Bekkers et al [21] | 5 | 5 | 4 | 5 | 2 | 3 | 4 | 4 | 32 (80) |
| Popescu et al [32] | 5 | 5 | 4 | 4 | 4 | 2 | 4 | 4 | 32 (80) |
| Sranacharoenpong et al [34] | 5 | 5 | 4 | 5 | 5 | 1 | 4 | 3 | 32 (80) |
| Chang et al [23] | 4 | 5 | 3 | 4 | 4 | 4 | 5 | 4 | 33 (82.5) |
| Moreira et al [19] | 5 | 5 | 4 | 5 | 5 | 0 | 5 | 5 | 34 (85) |
| Fontaine et al [26] | 5 | 5 | 3 | 5 | 5 | 2 | 5 | 5 | 35 (87.5) |
| Schneiderman et al [20] | 5 | 5 | 4 | 4 | 4 | 4 | 4 | 5 | 35 (87.5) |
| Cortese-Peske [25] | 5 | 5 | 4 | 5 | 4 | 5 | 5 | 5 | 38 (95) |
These elements give a useful starting point in describing a requisite information set for inclusion within all good quality research studies. The value of studies that fail to include and compliment these basic elements with additional standard or sufficient research information data is significantly reduced. The studies examined had low scores for the Ethics section overall. This section looked for information related to consideration of standard research ethics, such as participant ethics and researcher ethics, even where formal ethical approval had not been required. Information that conveys ethical considerations is a prerequisite of all research studies.
The relatively low scores achieved in relation to Design, Data Collection, and Results categories are also concerning, as this renders many of the studies difficult (if not impossible) to replicate. This section looked for inclusion of information on interventions, outcomes, or treatment measures, in addition to sufficient descriptions of the research design and rationale. A key requisite of effective TEL evaluation research is ensuring that the intervention is sufficiently described to support others, who may wish to make comparisons, or to confidently apply the research to their own practice or education program development [11]. Only in this way will we begin to establish a reliable evidence base around TEL evaluation. The higher average quality assessment scores were in relation to the Discussion section and Preliminary and Introductory elements such as title, abstract, background, and objectives information.
A subset of the studies [14,18-20] was randomly selected for additional scoring by PN. The studies were provided to two of the coauthors of this paper (SM and HVW) for them to provide an additional score using the CCAT scoring template and guide. The average difference between the original and additional scoring across 4 of the papers was found to be 19%.
Discussion
Principal Findings
Despite the growth in popularity and types of TEL education programs produced over the last two decades, this review was only able to identify a small pool of studies that met the inclusion criteria for TEL evaluation of a health care professional education program. Many of the included studies described the TEL methods evaluated as virtual learning environments, online or e-learning modules, platforms, or blended formats. There was little evidence provided within the selected studies regarding evaluation of bidirectional TEL approaches or newer types of TEL approaches such as Personal Learning Environments (PLEs). PLEs are activity spaces in which students interact and communicate with one another, and with experts, by using Web 2.0 tools. The ultimate result of using Web 2.0 tools is the development of collective learning approaches such as “just-in-time” and “at-your-fingertips” learning opportunities that can support a wide range of teaching and learning activities [38]. The evaluation studies identified in this review relied heavily on measuring TEL using learner satisfaction measures. Only one study [37] cited use of a structured approach to evaluating the more interactive two-way learning process between learner and tutor that is offered by models such as the Salmon-5 stage model [39]. The most widely cited types of learner outcome measurements used within educational evaluations are Kirkpatrick’s [40] models. While these methods of measurement may often provide a useful starting point for effective assessment of TEL evaluations, the model itself may not be ideally suited to the evaluation of TEL health care education. That is, where the Kirkpatrick model emphasizes increased confidence in newly acquired knowledge as being important, effective health care education would want evidence that this new knowledge is both learned and implemented in practice beyond a practitioner’s own perception of knowledge gain or confidence [13]. For health care education to truly support health care practice we need to be able to accurately measure the added knowledge, skills, or awareness that TEL programs may or may not provide.
This review of the literature concerning evaluations of TEL highlights the pattern that previous authors have noted for studies to employ a narrow focus in evaluation on either the technology equipment itself, measurement of learner satisfaction, or preknowledge and postknowledge scores [41,42]. Some of the studies included aspects of intralearner activity within their evaluation [18,22,27,28,30,33,37]. However, the dominance in the literature of evaluation studies measuring largely (or only) individual pretest and posttest knowledge presents a number of concerns within TEL for health care professionals. That is, where program assessments are relied upon to determine added value, learner gain, or improvement it is necessary to consider the extent to which they were matched to the actual TEL aims or enhancement being sought.
Testing methods can heavily influence the learner’s focus and how they approach learning. If we consider that one of the key functions of employing technology in learning is to help people connect more effectively with each other and the learning materials, and to inspire learner interaction in accordance with a social constructivist learning approach, then assessments that focus solely on the work of each individual (ie, cognitivist style, pretest and posttest scoring) may have a considerable impact on each learner’s behavior and the efficacy of the program overall [43]. There is an ongoing need for more TEL evaluation studies to detail the purpose of TEL interventions and the assessment and overall approaches adopted, and to demonstrate how the technology is enhancing the learning experience [2].
There is a need to be able to identify high quality TEL for health care education research studies and to be able to compare the outcomes from these sources to produce practical TEL evaluation tools. This need has also been highlighted throughout the last two decades in the literature on TEL in other contexts [44]. A more specific and standard approach to TEL evaluation in health care education with common measurements or tools would enable health care professionals, employers, and program designers to measure and then document effective TEL for health care education. In this way, a reliable TEL evidence base can grow and be progressively used to its full effect to influence or even transform educational programs for health care education now and in the future.
The CCAT tool used in this study enabled various forms of evidence presented within the literature to be explored in terms of TEL evaluation and the quality of the evidence presented. While some of the studies did present a full range of detailed research information, a number of those examined lacked information on the fundamental elements of good quality research. Ellaway [11] has highlighted similar issues related to published research for online learning where, as with TEL study interventions, they are often inadequately described. A more complex analysis of interassessor consistency in scoring using the CCAT tool could have been undertaken within this study. However, although the interassessor scoring was limited in this case by available resources, it was already building on the robust methods outlined by Crowe et al [17].
Limitations
This review has not measured standard educational quality parameters or set out to identify the requisite elements of a robust TEL evaluation guide (or tool) for health care professionals’ education. Instead, it focused on identifying what evidence of TEL evaluation for health care professional education already exists within the literature and examined whether it was of sufficient quantity and quality as an evidence base for organizations to use to develop increasingly effective and transformative TEL education programs. Although the subject of TEL dates to at least the 1990s in the context of further and higher education, its application within health care education is much more recent. On that basis, the decision was made to restrict the literature search to 2006-2017. Other databases such as Web of Science were checked but yielded few references that fully met the inclusion criteria, and such databases were therefore not included within the literature review methods for this study.
Conclusions
This review found limited published evidence of standard tools being implemented to measure TEL in health care education programs. Developing and implementing TEL health care education can require organizations to make considerable financial, human, and infrastructure investment. There is a mismatch between the scale of uptake of TEL in health care education and availability of a sufficiently robust evidence base of meaningful TEL evaluations in health care education. The outcomes of the systematic review and critical appraisal of this study support the views of Kirkwood and Price [2] who stated that there is a scarcity of published studies of practical TEL education programs that generate evidence that is appropriate to the interventions described, and that can be drawn upon.
A review of the TEL evaluation literature to help identify an evidence-based list of essential parameters to include within TEL health care education evaluation reports and studies would be a useful focus for further research. There continues to be a need to develop effective and standard TEL evaluation tools and for the publication of good quality studies that describe effective evaluation of TEL education for health care professionals . Studies often fail to provide sufficient detail to support transferability or direct future TEL health care education programs’ design and implementation. The use of a standard, practical, and quality approach to TEL evaluation, recording, and reporting with the same tools (or even parameters) across a variety of health care education programs would address this gap over time. This type of approach would also reduce duplication of efforts for organizations in creating or recreating tools, and importantly support cross-program and cross-organizational comparisons.
There is a range of guides, frameworks, and standards emerging in the literature and across practice to guide the design of TEL within health care and higher education institutions, programs, and resources. The models that have been proposed require widespread implementation, rigorous in-practice testing, and effective reporting to ensure that TEL education programs for health care professionals are evaluated in a more robust manner than is currently evident in the literature [13]. In this way, such programs can then usefully shape the emerging field of TEL evaluation for health care education.
Acknowledgments
The authors wish to acknowledge funding from National Health Service Highland and the University of the Highlands and Islands that supported this research study.
Abbreviations
- CCAT
Crowe Critical Appraisal Tool
- CPD
continuous professional development
- PICOS
Population, Intervention, Comparison, and Outcomes
- PLE
Personal Learning Environment
- TEL
technology-enhanced learning
Electronic database searches.
Summary of information from studies.
Footnotes
Conflicts of Interest: None declared.
References
- 1.Goodyear P, Retalis S. Technology-Enhanced Learning: Design Patterns and Pattern Language. Rotterdam, Netherlands: Sense Publishers; 2010. pp. 1–27. [Google Scholar]
- 2.Kirkwood A, Price L. Technology-enhanced learning and teaching in higher education: what is ‘enhanced’ and how do we know? A critical literature review. Learn Media Technol. 2013 Feb 20;39(1):6–36. doi: 10.1080/17439884.2013.770404. [DOI] [Google Scholar]
- 3.Carroll C, Booth A, Papaioannou D, Sutton A, Wong R. UK health-care professionals' experience of on-line learning techniques: a systematic review of qualitative data. J Contin Educ Health Prof. 2009;29(4):235–41. doi: 10.1002/chp.20041. [DOI] [PubMed] [Google Scholar]
- 4.Childs S, Blenkinsopp E, Hall A, Walton G. Effective e-learning for health professionals and students--barriers and their solutions. A systematic review of the literature--findings from the HeXL project. Health Info Libr J. 2005 Dec;22 Suppl 2:20–32. doi: 10.1111/j.1470-3327.2005.00614.x. doi: 10.1111/j.1470-3327.2005.00614.x.HIR614 [DOI] [PubMed] [Google Scholar]
- 5.The Higher Education Funding Council for England . Digital Education Resource Archive (DERA) England: Higher Education Funding Council for England (HEFCE); 2009. [2018-03-12]. Enhancing learning and teaching through the use of technology: a revised approach to HEFCE’s strategy for e-learning http://dera.ioe.ac.uk/140/ [Google Scholar]
- 6.Schostak J, Davis M, Hanson J, Schostak J, Brown T, Driscoll PS, Starke I, Jenkins N. 'Effectiveness of Continuing Professional Development' project: a summary of findings. Med Teach. 2010;32(7):586–92. doi: 10.3109/0142159X.2010.489129. [DOI] [PubMed] [Google Scholar]
- 7.Health Education England 2016. Mar, [2018-03-12]. Improving Safety Through Education and Training Report by the Commission on Education and Training for Patient Safety http://http/www.hee.nhs.uk/the-commission-on-education-and-training-for-patient-safety .
- 8.Lupton D. Digital sociology: an introduction. SSRN Journal. 2012 Aug 15;:1–16. doi: 10.2139/ssrn.2273418. https://papers.ssrn.com . [DOI] [Google Scholar]
- 9.Santy J, Smith L. Being an e-learner in health and social care: a student's guide. Abingdon, England: Routledge; 2007. pp. 1–33. [Google Scholar]
- 10.Ellaway R. Weaving the 'e's together. Med Teach. 2006 Nov;28(7):587–90. doi: 10.1080/01421590600909070.770734088 [DOI] [PubMed] [Google Scholar]
- 11.Ellaway R. Apples and architraves: a descriptive framework for e-learning research. Med Teach. 2010 Jan;32(1):95–7. doi: 10.3109/01421590903573966. [DOI] [PubMed] [Google Scholar]
- 12.Cook D, Ellaway RH. Evaluating technology-enhanced learning: a comprehensive framework. Med Teach. 2015;37(10):961–70. doi: 10.3109/0142159X.2015.1009024. [DOI] [PubMed] [Google Scholar]
- 13.Pickering JD, Joynes VCT. A holistic model for evaluating the impact of individual technology-enhanced learning resources. Med Teach. 2016 Dec;38(12):1242–1247. doi: 10.1080/0142159X.2016.1210112. [DOI] [PubMed] [Google Scholar]
- 14.Akroyd M, Jordan G, Rowlands P. Interprofessional, simulation-based technology-enhanced learning to improve physical health care in psychiatry: the recognition and assessment of medical problems in psychiatric settings course. Health Informatics J. 2016 Jun;22(2):312–20. doi: 10.1177/1460458214557098.1460458214557098 [DOI] [PubMed] [Google Scholar]
- 15.Whittemore R, Knafl K. The integrative review: updated methodology. J Adv Nurs. 2005 Dec;52(5):546–53. doi: 10.1111/j.1365-2648.2005.03621.x.JAN3621 [DOI] [PubMed] [Google Scholar]
- 16.Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res. 2014 Nov 21;14:579. doi: 10.1186/s12913-014-0579-0. https://bmchealthservres.biomedcentral.com/articles/10.1186/s12913-014-0579-0 .s12913-014-0579-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Crowe M, Sheppard L, Campbell A. Reliability analysis for a proposed critical appraisal tool demonstrated value for diverse research designs. J Clin Epidemiol. 2012 Apr;65(4):375–83. doi: 10.1016/j.jclinepi.2011.08.006.S0895-4356(11)00260-5 [DOI] [PubMed] [Google Scholar]
- 18.Lotrecchiano G, McDonald P, Lyons L, Long T, Zajicek-Farber M. Blended learning: strengths, challenges, and lessons learned in an interprofessional training program. Matern Child Health J. 2013 Nov;17(9):1725–34. doi: 10.1007/s10995-012-1175-8. [DOI] [PubMed] [Google Scholar]
- 19.Moreira I, Ventura SR, Ramos I, Rodrigues PP. Development and assessment of an e-learning course on breast imaging for radiographers: a stratified randomized controlled trial. J Med Internet Res. 2015 Jan 05;17(1):e3. doi: 10.2196/jmir.3344. http://www.jmir.org/2015/1/e3/ v17i1e3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Schneiderman J, Corbridge S, Zerwic JJ. Demonstrating the effectiveness of an online, computer-based learning module for arterial blood gas analysis. Clin Nurse Spec. 2009;23(3):151–5. doi: 10.1097/NUR.0b013e3181a075bc.00002800-200905000-00009 [DOI] [PubMed] [Google Scholar]
- 21.Bekkers M, Simpson SA, Dunstan F, Hood K, Hare M, Evans J, Butler CC, STAR Study Team Enhancing the quality of antibiotic prescribing in primary care: qualitative evaluation of a blended learning intervention. BMC Fam Pract. 2010 May 07;11:34. doi: 10.1186/1471-2296-11-34. https://bmcfampract.biomedcentral.com/articles/10.1186/1471-2296-11-34 .1471-2296-11-34 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Byrne E, Donaldson L, Manda-Taylor L, Brugha R, Matthews A, MacDonald S, Mwapasa V, Petersen M, Walsh A. The use of technology enhanced learning in health research capacity development: lessons from a cross country research partnership. Global Health. 2016 May 10;12(1):19. doi: 10.1186/s12992-016-0154-z. https://globalizationandhealth.biomedcentral.com/articles/10.1186/s12992-016-0154-z .10.1186/s12992-016-0154-z [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Chang W, Hsiao SS, Chang PC, Lee PH. Developing an E-learning education programme for staff nurses: processes and outcomes. Nurse Educ Today. 2008 Oct;28(7):822–8. doi: 10.1016/j.nedt.2008.02.003.S0260-6917(08)00021-X [DOI] [PubMed] [Google Scholar]
- 24.Chuo Y, Liu C, Tsai C. Effectiveness of e-learning in hospitals. Technol Health Care. 2015;23 Suppl 1:S157–60. doi: 10.3233/thc-150949.THC949 [DOI] [PubMed] [Google Scholar]
- 25.Cortese-Peske M. Icahn School of Medicine at Mount Sinai. ERIC; 2013. [2017-03-09]. Improving cultural competency and disease awareness among oncology nurses caring for adult T-cell leukemia and lymphoma patients http://www.proquest.com/en-US/products/dissertations/individuals.shtml . [Google Scholar]
- 26.Fontaine G, Cossette S, Heppell S, Boyer L, Mailhot T, Simard MJ, Tanguay JF. Evaluation of a Web-based E-learning platform for brief motivational interviewing by nurses in cardiovascular care: a pilot study. J Med Internet Res. 2016 Aug 18;18(8):e224. doi: 10.2196/jmir.6298. http://www.jmir.org/2016/8/e224/ v18i8e224 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Gill A. E-learning and professional development--never too old to learn. Br J Nurs. 2007;16(17):1084–8. doi: 10.12968/bjon.2007.16.17.27255. [DOI] [PubMed] [Google Scholar]
- 28.Goldberg GD, Clement DG, Cotter JJ. Development and alumni assessment of an interdisciplinary PhD program offered through a blended learning environment. J Allied Health. 2011;40(3):137–42. [PubMed] [Google Scholar]
- 29.Heartfield M, Morello A, Harris M, Lawn S, Pols V, Stapleton C, Battersby M. E-learning competency for practice nurses: an evaluation report. Aust J Prim Health. 2013;19(4):287–91. doi: 10.1071/PY13033.PY13033 [DOI] [PubMed] [Google Scholar]
- 30.Ingelbeen B, Aryeetey R, Khanna J, Mahamoudane N, Vincent D, Kiyan C, Maojo V, Lynen L, Zolfo M, Ramirez-Robles M. Exploring three approaches to offer distance learning courses through a social network of health researchers in three African countries. IEEE Xplore; International School of Tanganyika-Africa Conference Proceedings, 2014; 2014; Le Meridien Ile Maurice, Mauritius. IEE Explore; 2014. May 9, http/ieeexplore.ieee.org . [DOI] [Google Scholar]
- 31.Konstantinidis S, Kitsou A, Bamidis P, Mikos T, Zafrakas M, Nikolaidou M, Pappas C, Agorastos T. A SCORM compliant e-learning platform for cervical cancer pPrevention. Computer-Based Medical Systems, 2008; June 19, 2008; Jyvaskyla, Finland. IEEE Xplore: 2008. Jun 19, [DOI] [Google Scholar]
- 32.Popescu D, Avramescu ET, Pascual LJ, Pratikakis I, Perantonis S, Lupescu O. A Web-based e-training platform for biomedical engineering education. 2014 International Conference and Exposition on Electrical and Power Engineering; 2014; Iasi, Romania. 2014. Jun 19, http://ieeexplore.ieee.org/document/6969889/ [DOI] [Google Scholar]
- 33.Safwat A, Pourabdollah A. E-learning for healthcare professionals towards HIS in Egypt. 7th International Conference on Communications and Information Technology (ICICT); December 16, 2009; Cairo, Egypt. 2009. Dec 16, http://ieeexplore.ieee.org . [Google Scholar]
- 34.Sranacharoenpong K, Hanning RM, Sirichakwal PP, Chittchang U. Process and outcome evaluation of a diabetes prevention education program for community healthcare workers in Thailand. Educ Health (Abingdon) 2009 Dec;22(3):335.335 [PubMed] [Google Scholar]
- 35.Walsh K, Rafiq I, Hall R. Online educational tools developed by Heart improve the knowledge and skills of hospital doctors in cardiology. Postgrad Med J. 2007 Jul;83(981):502–3. doi: 10.1136/pgmj.2007.059501. http://europepmc.org/abstract/MED/17621623 .83/981/502 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Wang F. Valuation of Online Continuing Medical Education and Telemedicine in Taiwan. J Educ Techno Soc. 2008 Jan 01;11(4):190. https://pdfs.semanticscholar.org/c5e8/e8daa4b3cbdf5f6041ec173e99e8f75c4b41.pdf?_ga=2.229111580.1098686220.1523038790-1371194822.1509126301 . [Google Scholar]
- 37.Westbrook C. Online collaborative learning in health care education. EURODL. 2012 Jan 26;1:1–6. http://www.eurodl.org . [Google Scholar]
- 38.Rahimi E, van den Berg J, Veen W. A learning model for enhancing the student's control in educational process using Web 2.0 personal learning environments. Br J Educ Technol. 2014 May 13;46(4):780–792. doi: 10.1111/bjet.12170. [DOI] [Google Scholar]
- 39.Salmon G. E-moderating: the kiey to teaching and learning online. London: Routledge; 2004. Jan 22, [Google Scholar]
- 40.Kirkpatrick Partners. 2018. [2016-05-12]. The new world Kirkpatrick model https://www.kirkpatrickpartners.com/
- 41.Koehler N. Medical students’ use of and attitudes towards medical applications. Journal MTM. 2012 Dec 22;1(4):16–21. doi: 10.7309/jmtm.73. [DOI] [Google Scholar]
- 42.Wallace S, Clark M, White J. 'It's on my iPhone': attitudes to the use of mobile computing devices in medical education, a mixed-methods study. BMJ Open. 2012;2(4):2012–00109. doi: 10.1136/bmjopen-2012-001099. http://bmjopen.bmj.com/cgi/pmidlookup?view=long&pmid=22923627 .bmjopen-2012-001099 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Cubric M. Wiki-based process framework for blended learning. Proceeding of International Symposium on Wikis, 2007; International Symposium on Wikis; October 21-25, 2007; Montreal, Canada. 2007. Oct 25, pp. 11–24. [DOI] [Google Scholar]
- 44.Roblyer M. Educational technology research that makes a differenceries introduction. CITE. 2005;(Online serial, 5(2)):746–748. http://www.citejournal.org/volume-5/issue-2-05/seminal-articles/educational-technology-research-that-makes-a-difference-series-introductionDexter, [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Electronic database searches.
Summary of information from studies.